Method and Device for Situation Awareness

ABSTRACT

A method for situation awareness is provided. The method comprises: preparing a neural network trained by a learning set, wherein the learning set includes a plurality of maritime images and maritime information including object type information which includes a first type index for a vessel, a second type index for a water surface and a third type index for a ground surface, and distance level information which includes a first level index indicating that a distance is undefined, a second level index indicating a first distance range and a third level index indicating a second distance range greater than the first distance range; obtaining a target maritime image generated from a camera; and determining a distance of a target vessel based on the distance level index of the maritime information being outputted from the neural network which receives the target maritime image and having the first type index.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.16/557,859 filed on Aug. 30, 2019, which claims the benefit of U.S.Provisional Application Nos. 62/726,913 filed on Sep. 4, 2018 and62/741,394 filed on Oct. 4, 2018, and also claims priority to and thebenefit of Republic of Korea Patent Application Nos. 10-2018-0165857,10-2018-0165858 and 10-2018-0165859 all filed on Dec. 20, 2018, whichare incorporated by reference in their entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a situation awareness method and deviceusing image segmentation and more particularly, to a situation awarenessmethod and device using a neural network which performs imagesegmentation.

2. Description of the Related Art

Many accidents occur in vessel navigation, and the main cause thereof isknown to be ship operators' carelessness. When autonomous ships whichare not operated by humans and operate by themselves or unmanned shipsare commercialized, marine accidents are expected to be remarkablyreduced. To this end, global companies are carrying out autonomous shipdevelopment projects. Rolls-Royce are developing an autonomous ship incooperation with Google, and Yara and Kongsberg of Norway, Nippon Yusenof Japan, etc. are also working on autonomous ship development.

To complete an autonomous ship, it is necessary to be able to sense anobstacle without depending on visual inspection. Although various kindsof obstacle sensors are currently used, there remain limitations due tolow effectiveness. For example, an electronic chart display andinformation system (ECDIS) has limitations due to the inaccuracy of aglobal positioning system (GPS), update periods of an automaticidentification system (AIS), moving objects which have not beenregistered in the AIS, etc., and a radar has limitations due to thepresence of a non-observation region and noise. As a result, in order toaccurately detect an obstacle, it is necessary to check an obstaclevisually, which makes it difficult to complete an autonomous ship.

SUMMARY OF THE INVENTION

The present invention is directed to providing a situation awarenessmethod and device using a neural network which performs imagesegmentation.

The present invention is directed to providing a learning method of aneural network for situation awareness.

Objectives of the present invention are not limited to those mentionedabove, and objectives which have not been mentioned above should beclearly understood by those of ordinary skill in the technical field towhich the present invention pertains from this specification and theappended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will become more apparent to those of ordinary skill in theart by describing exemplary embodiments thereof in detail with referenceto the accompanying drawings.

FIG. 1 is a block diagram relating to an autonomous navigation methodaccording to an exemplary embodiment of the present invention.

FIG. 2 is a block diagram relating to a learning operation of a neuralnetwork according to an exemplary embodiment of the present invention.

FIG. 3 is a block diagram relating to an inference operation of a neuralnetwork according to an exemplary embodiment of the present invention.

FIG. 4 is a block diagram of a system which performs calculation relatedto situation awareness, autonomous navigation, etc. according to anexemplary embodiment of the present invention.

FIG. 5 is a block diagram of a ship sensor system according to anexemplary embodiment of the present invention.

FIGS. 6, 7, and 8 show examples of image segmentation according to anexemplary embodiment of the present invention.

FIG. 9 is a set of diagrams relating to data augmentation according toan exemplary embodiment of the present invention.

FIG. 10 is a set of diagrams relating to a neural network trainingmethod according to an exemplary embodiment of the present invention.

FIGS. 11 and 12 are diagrams relating to acquisition of locationinformation based on image pixels according to an exemplary embodimentof the present invention.

FIG. 13 is a block diagram relating to acquisition of final locationinformation according to an exemplary embodiment of the presentinvention.

FIGS. 14, 15, and 16 show examples of acquisition of final locationinformation according to an exemplary embodiment of the presentinvention.

FIGS. 17 and 18 are block diagrams relating to examples of a pathplanning operation according to an exemplary embodiment of the presentinvention.

FIG. 19 is a block diagram relating to an obstacle map update operationaccording to an exemplary embodiment of the present invention.

FIG. 20 is a set of diagrams relating to an obstacle map according to anexemplary embodiment of the present invention.

FIG. 21 is a block diagram relating to a location information conversionoperation according to an exemplary embodiment of the present invention.

FIG. 22 is a set of diagrams relating to an update region according toan exemplary embodiment of the present invention.

FIGS. 23 and 24 are diagrams relating to correction of locationinformation according to an exemplary embodiment of the presentinvention.

FIG. 25 is a set of diagrams relating to weight setting in whichmovement of an obstacle is taken into consideration according to anexemplary embodiment of the present invention.

FIG. 26 is a set of diagrams relating to weight setting in which sailingregulations are taken into consideration according to an exemplaryembodiment of the present invention.

FIG. 27 is a diagram relating to weight setting in which a buffer regionis taken into consideration according to an exemplary embodiment of thepresent invention.

FIGS. 28 and 29 are block diagrams relating to examples of an obstaclemap update operation according to an exemplary embodiment of the presentinvention.

FIG. 30 is a block diagram relating to a path generation operationaccording to an exemplary embodiment of the present invention.

FIG. 31 is a diagram relating to a path generation operation accordingto an exemplary embodiment of the present invention.

FIG. 32 is a block diagram relating to a path-following operationaccording to an exemplary embodiment of the present invention.

FIG. 33 is a diagram relating to visualization according to an exemplaryembodiment of the present invention.

FIGS. 34, 35, and 36 are block diagrams relating to examples ofautonomous navigation based on image segmentation according to anexemplary embodiment of the present invention.

FIG. 37 is a block diagram relating to a learning method of a neuralnetwork which outputs a control signal according to an exemplaryembodiment of the present invention.

FIG. 38 is a block diagram relating to an output of a neural networkaccording to an exemplary embodiment of the present invention.

FIG. 39 is a diagram relating to an index which represents objectsoverlapping each other according to an exemplary embodiment.

FIG. 40 is a diagram relating to an index representing connectivitybetween objects according to an exemplary embodiment.

FIG. 41 is a diagram relating to distinguishing between objects of thesame type based on distance information according to an exemplaryembodiment.

FIG. 42 is a set of diagrams relating to an index which is set so thattype information depends on distance information.

FIG. 43 is a diagram relating to distance information based on areference object according to an exemplary embodiment.

DETAILED DESCRIPTION

Exemplary embodiments described herein are intended to clarify thespirit of the present invention to those of ordinary skill in thetechnical field to which the present invention pertains. Accordingly,the present invention is not limited to the exemplary embodimentsdescribed herein, and the scope of the present invention should beconstrued as including modifications or alterations within the spirit ofthe present invention.

Terminology used herein has been selected from among general terms whichare widely used at the present time in consideration of functionality inthe present invention but may vary according to intentions of those ofordinary skill in the art to which the present invention pertains,precedents, the advent of new technology, or the like. However, when aspecific term is arbitrarily defined and used, the meaning of the termwill be described. Therefore, terms used herein should not simply bedefined literally but should be defined on the basis of the meaningsthereof and the overall content of the present invention.

Drawings in this specification are intended to readily describe thepresent invention. Shapes shown in the drawings may be exaggerated asnecessary to aid in understanding the present invention, and the presentinvention is not limited to the drawings. It will be also understoodthat, although the terms first, second, etc. may be used herein todescribe various elements, these elements should not be limited by theseterms. These terms are only used to distinguish one element fromanother. Thus, a first element could be termed a second element withoutdeparting from the technical spirit of the present invention.

When it is determined in this specification that detailed descriptionsof known configurations or functions related to the present inventionmay obscure the gist of the present invention, the detailed descriptionswill be omitted as necessary.

According to an aspect of the present invention, a method for learning aneural network performed by a computing means wherein the neural networkreceives a marine image and outputs information about a type and adistance of at least one object included in the marine image, the methodmay comprises: obtaining a marine image including a sea and an obstacle;obtaining a labelling data including a plurality of labelling valuesgenerated based on the marine image, wherein the plurality of labellingvalues includes at least one first labelling value corresponding to thesea and at least one second labelling value corresponding to theobstacle and determined by reflecting a type information and a distanceinformation of the obstacle; obtaining an output data including aplurality of output values by using a neural network, wherein the neuralnetwork receives the marine image and outputs the output data, and theplurality of output values includes a first output value correspondingto the first labelling value and a second output value corresponding tothe second labelling value; calculating an error value by using thelabelling data and the output data, wherein the error value iscalculated by considering the difference between the first labellingvalue and the first output value and the difference between the secondlabelling value and the second output value; and updating the neuralnetwork based on the error value; wherein the plurality of labellingvalues and the plurality of output values are selected from a pluralityof identification values, and at least one identification value of theplurality of identification values is determined by a combination ofinformation about a type and a distance of an object.

Herein, the error value may be calculated by considering the differencebetween the first labelling value and the first output value as a firstweight and the difference between the second labelling value and thesecond output value as a second weight, and the first weight maydecrease as the number of the first labelling value increases, and thesecond weight may decrease as the number of the second labelling valueincreases.

Herein, the marine image may further include a sky, the plurality oflabelling values may further include at least one third labelling valuecorresponding to the sky, the plurality of output values may furtherinclude a third output value corresponding to the third labelling value,and the error value may not change when the difference between the thirdlabelling value and the third output value changes.

Herein, the obtaining a labelling data may further include reflectingthe second labelling value to the first labelling value.

Herein, the first labelling value may not include a distance informationof the sea.

According to another aspect of the present invention, a method forlearning a neural network performed by a computing means wherein theneural network receives an image and outputs information about a typeand a distance of at least one object included in the image, the methodmay comprise: obtaining an image including a drivable region and anobstacle; obtaining a labelling data including a plurality of labellingvalues generated based on the image, wherein the plurality of labellingvalues includes at least one first labelling value corresponding to thedrivable region and at least one second labelling value corresponding tothe obstacle and determined by reflecting a type information and adistance information of the obstacle; obtaining an output data includinga plurality of output values by using a neural network, wherein theneural network receives the image and outputs the output data, and theplurality of output values includes a first output value correspondingto the first labelling value and a second output value corresponding tothe second labelling value; calculating an error value by using thelabelling data and the output data, wherein the error value iscalculated by considering the difference between the first labellingvalue and the first output value and the difference between the secondlabelling value and the second output value; and updating the neuralnetwork based on the error value; wherein the plurality of labellingvalues and the plurality of output values may be selected from aplurality of identification values, and at least one identificationvalue of the plurality of identification values may be determined by acombination of information about a type and a distance of an object.

According to another aspect of the present invention, a method forsituation awareness of a ship using a neural network performed by acomputing means, the method may comprise: obtaining a marine imageincluding a sailable region and an obstacle by a camera installed on aship, wherein the marine image includes a first pixel corresponding tothe sailable region and a second pixel corresponding to the obstacle;obtaining an output data by using a neural network performing imagesegmentation, wherein the neural network receives the marine image andoutputs the output data, and the output data includes a first outputvalue corresponding to the first pixel and a second output valuecorresponding to the second pixel and determined by reflecting a typeinformation of the obstacle and a first distance information of thesecond pixel; obtaining a second distance information of the secondpixel based on a location of the second pixel on the marine image and aposition information of the camera, wherein the position information ofthe camera is calculated by considering a position information of theship; and updating the second output value by reflecting a finaldistance information, wherein the final distance information may becalculated based on the first distance information and the seconddistance information.

Herein, the first distance information may be selected from a pluralityof categories and the second distance information may be represented bya distance value, wherein each of the plurality of categories mayinclude a distance range, and the updating the second output value maycomprise: determining an error based on whether the second distanceinformation is included in the first distance information; calculatingthe final distance information based on the error; and updating thesecond output value by reflecting the final distance information.

Herein, the calculating the final distance information may becalculating the first distance information as the final distanceinformation when the error occurs.

Herein, the calculating the final distance information may becalculating a distance from the second distance information to a maximumvalue of the first distance information as the final distanceinformation when the error does not occur.

According to another aspect of the present invention, an autonomousnavigation method of a ship based on a marine image and a neural networkmay include an operation of obtaining a training image including aplurality of pixel values and labeling data corresponding to thetraining image and including a plurality of labeling values which aredetermined by reflecting type information and distance information of atraining obstacle included in the training image, an operation in whichthe neural network receives the training image and outputs output dataincluding a plurality of output values corresponding to the labelingvalues, an operation of training the neural network using an errorfunction in which differences between the labeling values and the outputvalues are taken into consideration, an operation of obtaining themarine image from a camera installed on the ship, an operation ofobtaining type information and distance information of an obstacleincluded in the marine image using the neural network, the obstacleincluding a plurality of pixel values, an operation of obtainingdirection information of the obstacle included in the marine image onthe basis of locations of the pixel values of the obstacle on the marineimage, an operation of obtaining a location of the obstacle included inthe marine map on an obstacle map using the distance information and thedirection information of the obstacle included in the marine image, anoperation of generating the obstacle map using the location on theobstacle map, an operation of generating a following path followed bythe ship using the obstacle map and ship status information includinglocation information and position information of the ship, and anoperation of generating a control signal including a propeller controlsignal and a heading control signal using the following path so that theship may follow the following path.

Herein, the labeling values and the output values may be selected fromamong a plurality of identification values determined by combinations ofthe type information and the distance information of the obstacle.

Herein, distance information of the obstacle for determining theidentification values may include a plurality of categories each havinga distance range.

Herein, the control signal may be generated on the basis of at least oneof a control signal of a previous frame, tidal current, and wind.

Herein, the autonomous navigation method may further include anoperation of training a second neural network, which receives afollowing path and outputs a control signal, through reinforcementlearning, and the control signal may be generated by the second neuralnetwork on the basis of a control signal of a previous frame.

Herein, the operation of generating the obstacle map may further useinformation on movement of the obstacle included in the marine image.

Herein, the operation of obtaining the marine image may includeobtaining an image from the camera and generating the marine image bypreprocessing the image, and the preprocessing may be defogging of theimage.

According to another aspect of the present invention, an autonomousnavigation method of a ship based on a marine image and a neural networkmay include an operation of obtaining a training image including aplurality of pixel values and labeling data corresponding to thetraining image and including a plurality of labeling values which aredetermined by reflecting type information and distance information of atraining obstacle included in the training image, an operation in whichthe neural network receives the training image and outputs output dataincluding a plurality of output values corresponding to the labelingvalues, an operation of training a first neural network using an errorfunction in which differences between the labeling values and the outputvalues are taken into consideration, an operation of training a secondneural network which outputs a control signal using the type informationand the distance information of the training obstacle, an operation ofobtaining the marine image from a camera installed on the ship, anoperation of obtaining type information and distance information of anobstacle included in the marine image using the first neural network,the obstacle including a plurality of pixel values, an operation ofobtaining direction information of the obstacle included in the marineimage on the basis of locations of the pixel values of the obstacle onthe marine image, and an operation of generating a control signalincluding a propeller control signal and a heading control signal forthe ship by inputting the type information, the distance information,direction information, and ship status information including locationinformation and position information of the ship to the second neuralnetwork.

According to another aspect of the present invention, an autonomousnavigation method of a ship using end-to-end learning may include: anoperation of training a first neural network using training dataincluding images, location information, and position information andlabeling data including a propeller control signal and a heading controlsignal; an operation of obtaining a marine image from a camera installedon the ship; an operation of obtaining ship status information includinglocation information and position information of the ship; and anoperation of generating a control signal, which includes a propellercontrol signal and a heading control signal for the ship, using thefirst neural network, wherein the first neural network may include atleast one of object information of obstacles including distanceinformation of the obstacles, information on an obstacle map, andinformation on a path followed by the ship.

Herein, the autonomous navigation method may further include anoperation of obtaining at least one of the object information, theinformation on an obstacle map, and the information on a path followedby the ship from the first neural network.

According to another aspect of the present invention, a method ofprocessing an obstacle map related to a sailing path of a ship mayinclude an operation of obtaining an obstacle map including a pluralityof unit regions to which weights used for setting the sailing path areallocated, an operation of obtaining a marine image of the sea, anoperation of obtaining distance information and type information of anobstacle included in the marine image from the marine image using aneural network which is trained to output result values reflecting adistance and a type of object included in an image, an operation ofobtaining location information of the obstacle on the basis of thedistance information of the obstacle obtained from the neural networkand a direction in which the marine image has been captured, and anoperation of setting weights of first unit regions corresponding to azone within a sensing range of the marine image among the unit regions,wherein when the first unit regions are 1-1 unit regions correspondingto a zone detected in the marine image, the weights may be set on thebasis of the location information of the obstacle, and when the firstunit regions are 1-2 unit regions corresponding to a zone occluded bythe obstacle, the weights may be set on the basis of the typeinformation of the obstacle.

Herein, in the operation of setting weights of the first unit regions,when the 1-1 unit regions correspond to a location of the obstacle onthe basis of the location information of the obstacle, the weights maybe set to a first value reflecting unsuitability for sailing, and whenthe 1-1 unit regions do not correspond to the location of the obstacleon the basis of the location information of the obstacle, the weightsmay be set to a second value reflecting suitability for sailing.

Herein, in the operation of setting weights of the first unit regions,an area occupied by the obstacle may be determined according to the typeinformation of the obstacle, and the weights may be set to the firstvalue when the 1-2 unit regions correspond to the area occupied by theobstacle and may be set to a third value between the first value and thesecond value when the 1-2 unit regions do not correspond to the areaoccupied by the obstacle.

According to another aspect of the present invention, a method ofprocessing an obstacle map related to a sailing path may include: anoperation of obtaining an obstacle map including a plurality of unitregions to which weights reflecting sailing suitability used for settingthe sailing path are allocated in a range between a first valueindicating the maximum sailing suitability and a second value indicatingthe minimum sailing suitability; an operation of obtaining a marinevideo including a plurality of consecutive marine images; an operationof obtaining obstacle information related to a distance and a type ofobstacle included in the plurality of marine images from the pluralityof marine images using a neural network which is trained to outputresult values reflecting a distance and a type of object included in animage; an operation of obtaining, from a first marine image, theobstacle map in which the obstacle information obtained using the neuralnetwork has been applied to the weights of the plurality of unitregions; and an operation of updating the weights of the plurality ofunit regions using the obstacle information obtained using the neuralnetwork from a second marine image captured after the first marineimage, wherein when the plurality of unit regions corresponds to a zonewithin a sensing range of the second marine image, the weights may beupdated by considering the obstacle information related to the secondmarine image, and when the plurality of unit regions corresponds to azone not within the sensing range of the second marine image, theweights may be updated by considering the obstacle information relatedto the first marine image.

Herein, the operation of updating the weights by considering theobstacle information related to the second marine image may include anoperation of overriding the weights with values set according to theobstacle information related to the second marine image, and theoperation of updating the weights by considering the obstacleinformation related to the first marine image may include any one of afirst operation of adjusting the weights to a third value between thefirst value and the second value by considering the obstacle informationrelated to the first marine image and a second operation of maintainingthe weights.

Herein, the operation of updating the weights by considering theobstacle information related to the first marine image may include anoperation of performing the second operation on unit regionscorresponding to a stationary obstacle included in the first marineimage.

Herein, the zone within the sensing range of the second marine image mayinclude a zone detected in the second marine image and a zone notdetected in the second marine image, and the undetected zone may includea zone occluded by an obstacle included in the second marine image.

According to another aspect of the present invention, a method ofgenerating a sailing path in relation to obstacle avoidance of a shipmay include: an operation of obtaining an obstacle map including aplurality of unit regions to which weights used for generating thesailing path are allocated; an operation of obtaining a marine imageusing a camera installed on the ship; an operation of obtaining objectinformation, which includes type information and distance information,of an obstacle included in the marine image using a neural network towhich the marine image is input; an operation of obtaining directioninformation of the obstacle on the basis of a location of the obstacleon the marine image; an operation of obtaining location information ofthe obstacle on the obstacle map using the distance information and thedirection information; an operation of setting an update region of theobstacle map on the basis of a location and a direction of the camerawhich are dependent on a location and a direction of the ship,respectively; an operation of updating weights of unit regions includedin the update region using the location information and the obstaclemap; and an operation of generating a following path followed by theship by considering the weights and ship status information whichincludes location information and position information of the ship.

Herein, the object information may further include information on amovement direction of the obstacle, and the operation of updatingweights of the unit regions may include an operation of updating theweights of the unit regions included in the update region byadditionally using the information on the movement direction.

Herein, the operation of updating weights of the unit regions mayinclude an operation of updating the weights of the unit regionsincluded in the update region by additionally using information onsailing regulations.

Herein, the method may further include an operation of obtaining weatherenvironment information including at least one of tidal current andwind, and the operation of updating weights of the unit regions mayinclude an operation of updating the weights of the unit regionsincluded in the update region by additionally using the weatherenvironment information.

According to another aspect of the present invention, a method forsituation awareness may comprises: preparing a neural network trained bya learning set, wherein the learning set may include a plurality ofmaritime images and maritime information including object typeinformation which includes a first type index for a vessel, a secondtype index for a water surface and a third type index for a groundsurface, and distance level information which includes a first levelindex indicating that a distance is undefined, a second level indexindicating a first distance range and a third level index indicating asecond distance range greater than the first distance range, whereinpixels, in the plurality of maritime images, being labeled with thefirst type index, may be labeled with the second level index or thethird level index, and pixels, in the plurality of maritime images,being labeled with the first level index, may be labeled with the secondtype index or the third type index; obtaining a target maritime imagegenerated from a camera, the camera being installed on a port or avessel and monitoring surroundings thereof; and determining a distanceof a target vessel in the surroundings based on the distance level indexof the maritime information being outputted from the neural networkwhich receives the target maritime image and having the first typeindex.

Herein, the object type information may further include a fourth typeindex for a top surface of a vessel and a fifth type index for a sidesurface of a vessel, and pixels, in the plurality of maritime images,being labeled with the first type index, may be labeled with the thirdlevel index, and pixels, in the plurality of maritime images, beinglabeled with the fourth type index or the fifth type index, may belabeled with the second level index.

Herein, the object type information may further include a fourth typeindex for a top surface of a vessel and a fifth type index for a sidesurface of a vessel, and pixels, in the plurality of maritime images,being labeled with the first type index and the second level index, maybe labeled with one of the fourth type index and the fifth type index.

Herein, the object type information may further include a fourth typeindex for a crane, and pixels, in the plurality of maritime images,being labeled with the fourth type index, may be labeled with the firsttype index or the third type index.

Herein, the method may further comprise: detecting a water surface inthe surroundings, using the neural network, from the target maritimeimage, according to the object type information of the maritimeinformation being outputted from the neural network based on the targetmaritime image and having the second type index; and determining adistance to the detected water surface as an undefined distance.

Herein, the determining may comprise ignoring the distance levelinformation of the maritime information being outputted from the neuralnetwork based on the target maritime image and having the second typeindex.

Herein, the maritime information may further include an objectidentifier, and pixels, in the plurality of maritime images, beinglabeled with the first type index and the second level index, may belabeled with the object identifier.

Herein, the pixels being labeled with the first type index and thesecond level index may include one or more first pixels and one or moresecond pixels, and the object identifier labeled to the first pixels maybe different from the object identifier labeled to the second pixels.

Herein, the method may further comprise: generating an obstacle mapbased on the measured distance of the vessel, wherein the target vesselis reflected on the obstacle map when the outputted distance level indexcorresponds to the second level index.

Herein, the determining may comprise obtaining distance information ofthe target vessel based on at least one of a location of the targetvessel on the target maritime image and a position information of thecamera, and the distance of the target vessel may be determined furtherbased on the obtained distance information.

Herein, the obtained distance information may be represented by adistance value, the determining further may comprise determining anerror based on whether the obtained distance information is included inthe outputted distance level index, and the distance of the targetvessel may be determined further based on the error.

Herein, when the error occurs, the distance of the target vessel may bedetermined as a distance range corresponding to the outputted distancelevel index, and when the error does not occur, the distance of thetarget vessel may be determined as a distance range from the distancevalue to a maximum value of the distance range corresponding to theoutputted distance level index.

According to another aspect of the present invention, a device forsituation awareness may comprises: a memory for storing a neural networktrained by a learning set, wherein the learning set may include aplurality of maritime images and maritime information including objecttype information which includes a first type index for a vessel, asecond type index for a water surface and a third type index for aground surface, and distance level information which includes a firstlevel index indicating that a distance is undefined, a second levelindex indicating a first distance range and a third level indexindicating a second distance range greater than the first distancerange, wherein pixels, in the plurality of maritime images, beinglabeled with the first type index, may be labeled with the second levelindex or the third level index, and pixels, in the plurality of maritimeimages, being labeled with the first level index, may be labeled withthe second type index or the third type index; and a controllerconfigured to: obtain a target maritime image generated from a camera,the camera being installed on a port or a vessel and monitoringsurroundings thereof; and determine a distance of a target vessel in thesurroundings based on the distance level index of the maritimeinformation being outputted from the neural network which receives thetarget maritime image and having the first type index.

According to another aspect of the present invention, a method forsituation awareness may comprises: preparing a neural network trained bya learning set, wherein the learning set may include a plurality ofmaritime images and maritime information including: object typeinformation which includes a first type index for a vessel and a secondtype index for a water surface, distance level information whichincludes a first level index indicating that a distance is undefined, asecond level index indicating a first distance range and a third levelindex indicating a second distance range greater than the first distancerange, and surface type information which includes a first surface indexfor a side surface of a vessel and a second surface index for a topsurface of a vessel, wherein the first surface index and the secondsurface index may be only labeled to pixels being labeled with the firsttype index and the second level index; obtaining a target maritime imagegenerated from a camera which is installed on a port or a vessel andmonitors surroundings thereof; detecting the vessel in the surroundingsbased on the object type information of the maritime information beingoutputted from the neural network which receives the target maritimeimage and having the first type index; determining a distance to thedetected vessel based on the distance level information of the maritimeinformation being outputted from the neural network which receives thetarget maritime image and having the first type index; and detecting aboundary of the detected vessel based on the surface type information ofthe maritime information being outputted from the neural network whichreceives the target maritime image and having the first type index andthe second level index.

Herein, the boundary of the detected vessel may correspond to a boundarybetween a side surface of the detected vessel and a top surface of thedetected vessel.

According to another aspect of the present invention, a method formeasuring a distance may comprise: preparing a neural network trained bya learning set including a plurality of images and object informationlabeled to pixels of the plurality of images, wherein at least a portionof the plurality of images may include at least a portion of a referenceobject being located at a lower portion of the at least the portion ofthe plurality of images, a moving object and a drivable area, and theobject information may reflect an object type and a relative proximitylevel from the reference object, wherein pixels corresponding to themoving object may be labeled with the object information of which theobject type indicates the moving object and of which the relativeproximity level indicates a corresponding section where the movingobject is disposed among a plurality of sections dividing the drivablearea, wherein the plurality of sections may be sequentially arrangedfrom a borderline between the drivable area and the reference object toan upper boundary of the drivable area, and wherein pixels correspondingto the drivable area may be labeled with the object information of whichthe object type indicates the drivable area and of which the relativeproximity level indicates that a proximity is undefined; obtaining atarget image; calculating the object information related to the targetimage from the target image by using the neural network; detecting themoving object in the target image based on the calculated objectinformation; and determining a proximity of the moving object in thetarget image based on the relative proximity level being reflected bythe calculated object information.

Herein, the relative proximity level may include a first proximity leveland a second proximity level indicating further proximity than the firstproximity level.

Herein, the target image may be captured by a camera which is installedon a port or a vessel, and the reference object is one of the port orthe vessel where the camera is installed.

Herein, one or more boundaries between two adjacent sections may beformed considering a shape of the reference object.

Hereinafter, an autonomous navigation method and device using imagesegmentation according to an exemplary embodiment of the presentinvention will be described.

For autonomous navigation of a moving object according to an exemplaryembodiment of the present invention, the moving object may performsituation awareness, path planning, and path-following operations. Themoving object may determine surrounding obstacles and/or a navigableregion through situation awareness, generate a path on the basis of thesurrounding obstacles and/or navigable region, and then travel by itselfalong the generated path.

FIG. 1 is a block diagram relating to an autonomous navigation methodaccording to an exemplary embodiment of the present invention. Referringto FIG. 1, a moving object may obtain object information through asituation awareness operation S1000, obtain a following path on thebasis of the obtained object information through a path-planningoperation S3000, generate a control signal on the basis of the obtainedfollowing path through a path-following operation S5000, and travelautonomously.

The aforementioned operations may be consecutively performed forautonomous navigation of the moving object. One cycle of the situationawareness operation S1000, the path-planning operation S3000, and thepath-following operation S5000 will be described as one frame.

Among the types of autonomous navigation of moving objects, autonomousnavigation of a ship will be mainly described below. However, autonomousnavigation is not limited to a ship and may be applied to various movingobjects, such as a car and a drone.

Autonomous navigation may be performed using a neural network. A neuralnetwork is an algorithm modeled after the neural network structure ofhuman brain. A neural network may include one or more layers includingone or more nodes or neurons, and the nodes may be connected throughsynapses. Data input to a neural network (input data) may be passedthrough synapses and output through nodes (output data), and informationmay be obtained accordingly.

The types of neural networks include a convolutional neural network(CNN) which extracts features using a filter and a recurrent neuralnetwork (RNN) which has a structure in which an output of a node is fedback as an input. Besides, there are various types of neural networks,such as a restricted Boltzmann machine (RBM), a deep belief network(DBN), a generative adversarial network (GAN), and a relation network(RN).

Before using a neural network, it is necessary to train the neuralnetwork. Alternatively, it is possible to train a neural network whileusing another neural network. An operation of training a neural networkwill be referred to as a training operation, and an operation of usingthe neural network will be referred to as an inference operation.

There are various learning methods of a neural network, such assupervised learning, unsupervised learning, reinforcement learning, andimitation learning. Reinforcement learning may be referred to as aMarkov decision process. Alternatively, reinforcement learning may referto a method for an agent to maximize a reward in a certain environment.

FIG. 2 is a block diagram relating to supervised learning as an exampleof a learning operation of a neural network according to an exemplaryembodiment of the present invention. Referring to FIG. 2, an untrainedneural network may receive training data, output output data, comparethe output data with labeling data, backpropagate the error, and thus betrained. The training data may include an image, and the labeling datamay include ground truth. Alternatively, the labeling data may be datagenerated by a user or a program.

FIG. 3 is a block diagram relating to an inference operation of a neuralnetwork according to an exemplary embodiment of the present invention.Referring to FIG. 3, a trained neural network may receive input data andoutput output data. Information which is inferable in the inferenceoperation may vary according to information of training data in alearning operation. Also, the accuracy of the output data may varyaccording to the degree of learning of the neural network.

FIG. 4 is a block diagram of a system 10 which performs calculationrelated to situation awareness, autonomous navigation, etc. according toan exemplary embodiment of the present invention. The system 10 mayperform training and/or inference of a neural network.

Referring to FIG. 4, the system 10 may include a control module 11, aninput module 13, and an output module 15.

The control module 11 may perform a neural network's learning,inference, image segmentation, situation awareness based thereon, pathplanning, such as obstacle map update and path generation, controlsignal generation for path following, autonomous navigation, and thelike. Also, it is possible to perform an operation of receiving variousinputs through the input module 13, an operation of outputting variousoutputs through the output module 15, and an operation of storingvarious data in a memory or obtaining various data from the memory underthe control of the control module 11. Various operations or steps setforth herein as embodiments may be construed as being performed by thecontrol module 11 or under the control of the control module 11 unlessspecifically mentioned.

The input module 13 may receive information from the outside of thesystem 10. As an example, the input module 13 may be a ship sensorsystem which will be described below but is not limited thereto.

The output module 15 may output results of calculation and the likeperformed by the control module 11. For example, the output module 15may output an image, such as a marine image, image segmentation results,a control signal, and the like. For example, the output module 15 may bea display, a signal output circuit, and the like but is not limitedthereto.

The input module 13 and the output module 15 may be separate modules butmay be implemented as one module. For example, the input module 13 andthe output module 15 may be implemented as one integrated module, whichmay receive information from the outside and output results ofcalculation and the like performed by the control module 11.

The control module 11, the input module 13, and the output module 15 mayinclude a controller. The controller may process and calculate variousinformation and control components of the modules. The controller may bephysically provided in the form of an electronic circuit which processesan electrical signal. The modules may physically include only a singlecontroller but, to the contrary, may include a plurality of controllers.As an example, the controller may be one or more processors installed inone computing means. As another example, the controller may be providedas processors which are installed on a server and a terminal physicallyseparate from each other and cooperate with each other throughcommunication.

The control module 11, the input module 13, and the output module 15 mayinclude a communicator. The modules may transmit and receive informationthrough the communicator. As an example, the input module 13 maytransmit information obtained from the outside through the communicator,and the control module 11 may receive the information transmitted by theinput module 13 through the communicator. As another example, thecontrol module 11 may transmit a calculation result through thecommunicator, and the output module 15 may receive the informationtransmitted by the control module 11 through the communicator. Thecommunicator may perform wired or wireless communication. Thecommunicator may perform bidirectional or unidirectional communication.

The control module 11, the input module 13, and the output module 15 mayinclude a memory. The memory may store various processing programs,parameters for processing of the programs, the processing result data,or the like. For example, the memory may store data required fortraining and/or inference, a neural network in which learning is inprogress or completed, and the like. The memory may be implemented as anon-volatile semiconductor memory, a hard disk, a flash memory, a randomaccess memory (RAM), a read only memory (ROM), an electrically erasableprogrammable read-only memory (EEPROM), a tangible non-volatilerecording medium, or the like.

The system 10 shown in FIG. 4 is only an example, and a configuration ofthe system 10 is not limited thereto.

When a neural network is used for autonomous navigation, referring toFIG. 1, the situation awareness operation S1000, the path-planningoperation S3000, or the path-following operation S5000 may be performedby the neural network. Alternatively, two or more of the aforementionedoperations may be performed by the neural network, or one thereof may beperformed by a plurality of neural networks. When a plurality of neuralnetworks is used, the neural networks may be connected in parallel orseries. For example, when the neural networks are connected in series,an output of one neural network may be an input to another neuralnetwork.

The situation awareness operation, the path-planning operation, and thepath-following operation will be described in detail below. In order toperform an autonomous navigation method, it is unnecessary to performall of the following operations. Only some of the operations may beperformed, and a specific operation may be performed repeatedly.

For sailing of a ship, it is necessary to find obstacles and/or asailable region through situation awareness, and this is the same forautonomous navigation. Obstacles refer to all objects which hindersailing such as geographical features, structures, ships, buoys, andpeople. Situation awareness is not only to detect an obstacle but alsoto obtain information on surroundings of the ship. Detecting an obstacleis to obtain information about whether there is an obstacle, the type ofobstacle, a location of an obstacle, and the like.

FIG. 5 is a block diagram of a ship sensor system according to anexemplary embodiment of the present invention. Referring to FIG. 5,during sailing of a ship, an obstacle detection sensor, such as radar,light detection and ranging (LiDAR), or an ultrasonic detector, and anautomatic identification system (AIS) are used in general.

In addition, it is possible to detect an obstacle using a camera.Referring to FIG. 5, examples of a camera include a monocular camera, abinocular camera, an infrared (IR) camera, and a time-of-flight (TOF)camera but are not limited thereto.

As a method of detecting an obstacle with an image obtained from acamera, image segmentation may be used. A situation awareness operationperformed using image segmentation will be described in detail below.

Image segmentation may be to divide an image into regions according toan attribute. Alternatively, image segmentation may be a process ofallocating an attribute value to each pixel of an image. For example,the attribute may be a type of object. In this case, image segmentationmay divide an object included in an image into pixels. Alternatively,image segmentation may indicate which object corresponds to a specificpixel.

Image segmentation may be performed using a neural network. One neuralnetwork may be used, or a plurality of neural networks may separatelyperform image segmentation, and the results may be combined forsituation awareness. As the network structure of a neural network forimage segmentation, various structures, such as an efficient neuralnetwork (ENet) and its variations, may be used.

The case of performing the situation awareness operation S1000 of FIG. 1through image segmentation based on a neural network will be describedin detail below.

A neural network which performs image segmentation to sense surroundingsmay receive an image and output object information. Referring to FIGS. 2and 3, training data and input data may be in the form of an image,which may include a plurality of pixels. Output data and labeling datamay be object information. Additionally, it is possible to visuallydeliver information by visualizing output data and labeling data.

Any kind of image may be used to perform image segmentation. The imagemay be an image captured by a camera. Referring to FIG. 5, it ispossible to use images obtained from various cameras, such as amonocular camera, a binocular camera, an IR camera, and a TOF camera.Also, the image is not limited to a two-dimensional (2D) image and maybe a three-dimensional (3D) image or the like.

When image segmentation is performed once, only one image may be input.

Alternatively, a plurality of images may be input.

The image captured by the camera may be preprocessed and then input tothe neural network. Preprocessing refers to any kind of processingperformed on a captured image and may include image normalization, imageresizing, cropping, noise removal, image defogging, fine dust removal,salt removal, waterdrop removal, and combinations thereof. As an exampleof preprocessing, defogging may be converting an image of a foggy areainto an image of a clear area through preprocessing. As another exampleof preprocessing, normalization may be averaging red, green, blue (RGB)values of entire pixels of an RGB image and subtracting the average fromthe RGB image. Waterdrop removal may be removing waterdrops from animage showing waterdrops on the lens of a camera through preprocessing,removing raindrops from an image captured on a rainy day throughpreprocessing, and the like. Performance and/or accuracy of the neuralnetwork may be improved through such image preprocessing.

A method of preprocessing an image and then inputting the preprocessedimage to a neural network has been described above. However, a neuralnetwork including a preprocessing operation may be trained and used. Forexample, a neural network may be trained to output a result as if animage of a clear area was input even when an image of a foggy area isinput to the neural network.

Output data and/or labeling data of the neural network may correspond toan input image and/or a training image, respectively. Also, output dataand labeling data may correspond to each other.

Output data and labeling data may include a plurality of pieces ofinformation. Alternatively, output data and labeling data may bedetermined on the basis of a plurality of pieces of information.

Output data and labeling data may include a plurality of output valuesand a plurality of labeling values, respectively. The output values andthe labeling values may include a plurality of pieces of information.Alternatively, the output values and the labeling values may bedetermined on the basis of a plurality of pieces of information.

Output values and labeling values may correspond to pixels of an imageinput to the neural network. Also, the output values and the labelingvalues may correspond to each other.

The number of output values and the number of labeling values areidentical to the number of pixels of an input image. For example, when a256×256 pixel image is input, the number of output values and the numberof labeling values may be 256×256=65,536. Alternatively, the number ofoutput values and the number of labeling values may be n times (n is aninteger) the number of pixels of an input image. A case in which thenumber of output values and the number of labeling values are identicalto the number of pixels of an input image will be mainly described.However, the present invention is not limited to this case, and thenumber of pixels, the number of output values, and the number oflabeling values may differ from each other.

Output data and labeling data may be object information. The term“object information” refers to information on an attribute of an objectincluded in an image, and any information relating to the object may bethe object information.

Objects may include obstacles, such as geographical features,structures, ships, buoys, and people, a zone in which a ship may sail,such as the ocean, and a zone which may not be related to sailing of aship, such as the sky. The information may be any kind of information,such as whether there is an object, a type of object, the location of anobject, the movement direction of an object, the movement speed of anobject, and navigation mark information. The navigation mark informationmay include a lateral mark, cardinal marks, an isolated danger mark, asafe water mark, a special mark, and other marks. The location of anobject may include the distance and the direction to the object and maybe a relative location or an absolute location. Object information mayinclude only some or all of the objects and the pieces of information.For example, information on an obstacle among objects may be referred toas obstacle information.

Among the pieces of object information, location information will bedescribed in detail below.

When location information is learned in the training operation of aneural network, it is possible to infer location information. Thelocation information is information related to the location of anobject. For example, location information may be a distance and/or adirection. Alternatively, location information may be coordinates atwhich an object is present.

Distance information of an object may be the distance from a camera,which captures an image of the object, to the object. Alternatively,distance information of an object may be the distance from an arbitrarypoint to the object. Direction information of an object may be the anglebetween the object and a camera which captures an image of the object.Alternatively, direction information of an object may be the anglebetween an arbitrary point and the object.

Location information may be a relative location of an object or anabsolute location of the object.

It is possible to obtain location information of an object included inan image through image segmentation. Alternatively, it is possible toobtain location information of pixels or pixel groups in an image. FIG.6 is a diagram relating to image segmentation according to an exemplaryembodiment of the present invention. Referring to FIG. 6, output data130 in which an input image 110 has been divided according to a specificattribute may be obtained through image segmentation. The input image110 includes objects including two ships, the sea, and the land, and theoutput data 130 in which the objects are discriminated according totypes and distances may be obtained by inputting the input image 110 tothe neural network. For example, the two ships having different piecesof distance information may be shown to be different in the output data130.

FIG. 6 is an example of a location information presentation method basedon image segmentation. Location information may be also presented invarious other ways. Location information may include only distanceinformation or may include other object information, such as directioninformation, whether there is an object, and a type of object, inaddition to the distance information.

FIGS. 7 and 8 are diagrams relating to image segmentation according toan exemplary embodiment of the present invention. Exemplary embodimentsfor presenting location information will be described below withreference to FIGS. 7 and 8, but the presentation method is not limitedthereto.

Location information may be presented as a plurality of categorieshaving a certain range. For example, distance information may bepresented as short distance, middle distance, long distance, and thelike, and direction information may be presented as a left direction, aforward direction, a right direction, and the like. It is also possibleto combine location information and direction information and presentthe combination as short distance to the left side, long distance to theright side, and the like.

The category may be presented in numbers. For example, among the piecesof distance information, the short distance, the middle distance, andthe long distance may be presented as −1, 0, and +1, respectively.Referring to FIG. 7, when images of 256×256 pixels are used as trainingimages and the input image 110, output data and labeling data may be256×256 matrices, and each element of the matrices may have any onevalue of −1, 0, and +1 according to distance information.

Location information may be presented with an actual distance value anddirection value. For example, distance information may be presented inunits of meters, and direction information may be presented in units ofdegrees. Referring to FIG. 7, each element of output data and labelingdata matrices presenting distance information may have a distance valuein units of meters, such as 10 m and 15 m.

Location information may be presented by converting an actual distancevalue and direction value into values in a certain range. Locationinformation may be uniformly normalized or may be non-uniformlynormalized using a log function or an exponential function. For example,distance information may be normalized into a value of 0 to 1. Referringto FIG. 7, each element of output data and labeling data matricespresenting distance information may have a value of 0 to 1.

Location information may be presented as an RGB value. In this case,output data and labeling data become RGB images such that a user mayeasily recognize location information without a visualization operation.Referring to FIG. 8, a ship and geographical features included inlabeling data and the output data 130 may correspond to different RGBvalues due to different pieces of distance information and may bepresented in different colors. In comparison to FIG. 7, the labelingdata and the output data 130 may be 256×256×3 matrices in a matrixrepresentation.

Location information and other object information may be presentedtogether. Object information other than location information is referredto as additional information below.

Additional information may be presented independently of locationinformation. Alternatively, location information and additionalinformation may be presented simultaneously.

When location information and additional information are independentlypresented, the location information and the additional information maybe presented as separate data sets. For example, when a 256×256 pixelimage is input to the neural network, location information may be a256×256 matrix, and additional information may be n 256×256 matrices. nmay be the number of pieces of additional information.

When location information and additional information are presentedsimultaneously, the location information and the additional informationmay be presented as one data set. For example, location information anda type of obstacle may be presented together using the above-describedmethod of presenting location information in a plurality of categorieshaving a certain range. Table 1 shows a presentation method in whichdistance information and a type of obstacle are taken into considerationtogether according to an exemplary embodiment of the present invention.Referring to Table 1, it is possible to set classes by consideringdistance information and types of obstacles together and allocate anidentification value to each of the classes. For example, anidentification value of 1 may be allocated by considering short distancewhich is distance information and geographical features which isobstacle type information. Referring to Table 1 and FIG. 7, each elementof output data and labeling data matrices may have an identificationvalue of 1 to 10. Table 1 shows an example in which distance informationand obstacle type information are taken into consideration together, andit is also possible to consider other additional information, such asdirection information, obstacle movement directions, speed, andnavigation marks, together. It is unnecessary for all identificationvalues to correspond to a plurality of pieces of information or the sametype of information. For example, a specific identification value maycorrespond to only distance information, another identification valuemay correspond to only obstacle type information, and anotheridentification value may correspond to both distance information andobstacle type information. As such, identification values may bepresented in various ways according to circumstances.

TABLE 1 Identification value Class 1 Short distance + Geographicalfeatures 2 Middle distance + Geographical features 3 Long distance +Geographical features 4 Short distance + Stationary obstacle 5 Middledistance + Stationary obstacle 6 Long distance + Stationary obstacle 7Short distance + Moving obstacle 8 Middle distance + Moving obstacle 9Long distance + Moving obstacle 10 Others

For the accuracy of a neural network, the learning process is important.A neural network may be trained in various ways, such as supervisedlearning, unsupervised learning, reinforcement learning, and imitationlearning, without limitation.

To improve the accuracy of a neural network, it may be necessary toobtain various training images. When a neural network is trained usingvarious training images, it is possible to handle various input images.For example, foggy training images may be required to obtain informationfrom foggy images through image segmentation. Alternatively, it may benecessary to use images of a rainy day as training images in order toobtain accurate image segmentation results from a rainy day image.

In the case of a marine image, it may be difficult to obtain trainingimages because it is necessary to take a picture on a ship in the sea.In particular, it may be more difficult because it is necessary toconsider various marine situations for the accuracy of a neural network.Marine situations may include sea fog, fog, sea clutter, snow, rain, andthe like.

In addition to the method of obtaining training images on a ship in thesea, it is possible to obtain training images through data augmentation.Data augmentation refers to all methods other than a method of obtainingtraining images by directly taking pictures. As an example, a trainingimage may be obtained by superimposing fog on an image of a clear daythrough a program or the like. As another example, training images inwhich different sea colors are taken into consideration according toregions may be generated by arbitrarily changing a sea color. FIG. 9 isa set of diagrams relating to data augmentation according to anexemplary embodiment of the present invention. Referring to FIG. 9, itis possible to obtain images of a foggy day, a rainy day, and a foggyrainy day from a marine image of a clear day.

There are some cases of requiring labeling data corresponding to atraining image according to a learning method of a neural network. Forexample, supervised learning may require labeling data. Labeling datamay be generated from a training image.

Labeling data may include various kinds of information according to thepurpose of training a neural network. As an example, labeling data mayinclude location information corresponding to each of pixels and/orpixel groups of an image or location information corresponding to eachof regions of an image. Alternatively, labelling data may includelocation information of an object included in an image. Since a labelingdata presentation method varies according to the aforementioned locationinformation presentation methods, it may be necessary to generatelabeling data by considering a corresponding location informationpresentation method.

Obtaining labelling data including distance information will bedescribed in further detail. Distance information may be obtained usinga depth camera. The depth camera may be a stereo type, a structuredpattern type, a TOF type, or the like or may be a combination of two ormore thereof. It is possible to generate one piece of labelling data byobtaining distance information of each pixel in an image from the depthcamera. In addition to this, various methods for obtaining distanceinformation may be used.

Labelling data may be generated to include additional information aswell as distance information. When labelling data is generated toseparately include distance information and additional information, theabove-described method may be used without change. When distanceinformation and additional information are taken into considerationtogether and presented as one value, modified data in which distanceinformation obtained through a depth camera or the like and additionalinformation are taken into consideration together may be labelling dataas shown in Table 1.

Labelling data may be passively generated by a user. Alternatively,labelling data may be generated by a neural network or another algorithmfor generating labelling data. For example, an output of a first neuralnetwork which has been trained already may be used as labelling data ofa second neural network. FIG. 10 is a set of diagrams relating to amethod of training a second neural network using an output of a firstneural network as labelling data according to an exemplary embodiment ofthe present invention. Referring to FIG. 10, a first neural networkwhich receives a visible light image and performs image segmentation mayhave been trained, and a second neural network which receives an IRlight image and performs image segmentation may have not been trained.When a visible image and an IR image of the same scene are obtained andinput to the first and second neural networks, respectively, an outputof the first neural network may be accurate segmentation results, and anoutput of the second neural network may be inaccurate segmentationresults. The output of the first neural network may be compared with theoutput of the second neural network which is obtained using the outputof the first neural network as labelling data of the second neuralnetwork, and the error may be backpropagated to the second neuralnetwork such that the second neural network may be trained.

Automatically generated information may be corrected thereafter and usedas labelling data. For example, after pixel-specific distanceinformation obtained from a depth camera is corrected, labelling datamay be generated on the basis of the corrected data. Corrected data maybe more accurate values. For example, corrected data of distanceinformation may be more accurate distance values.

It is possible to generate labelling data by intentionally distortinginformation. For example, distance information obtained from a depthcamera may be intentionally distorted and applied to labelling data. Asmall obstacle, such as a buoy, may be labelled as having a size greaterthan the actual size. For this reason, a neural network may be trainedto detect a small obstacle well.

Another method for improving the accuracy of a neural network is toadopt an appropriate error backpropagation method. An error (loss) is ameasure of the accuracy of a neural network and may refer to thedifference between output data of the neural network and labelling data.

A total error of output data may be calculated using errors of outputvalues of the output data. Alternatively, a total error of output datamay be presented as a function of errors of output values of the outputdata. An error may be defined in various ways, such as a mean squareerror or a cross entropy error. For example, a total error may be thesum of errors of output values. Alternatively, a total error may becalculated by applying certain values (weights) to errors of outputvalues. As an example, the total error may be calculated by adding upproducts of the weights and the errors of the output values.

A total error may be calculated using information included in outputdata. For example, the total error may be calculated using typeinformation of an object included in the output data. Expression 1 belowis an example of a total error calculated using type information of anobject according to Table 1.

$\begin{matrix}{{mean}\mspace{11mu} \left( {\sum\limits_{l}{E_{c} \times W_{class}}} \right)} & {{Expression}\mspace{14mu} 1}\end{matrix}$

In Expression 1,

${W_{class} = \frac{1}{\ln \left( {c + P_{class}} \right)}},$

Pclass is the number of pixels constituting the object, c is a hyperparameter, and Ec is an identification value.

Errors of all output values may be taken into consideration to calculatea total error. Alternatively, to calculate a total error, errors of onlysome output values may be taken into consideration, and errors of otheroutput values may not. For example, when output values includeinformation on the types of objects, errors of only some objects may betaken into consideration, and errors of other objects may not. Theobjects taken into consideration may be objects involved in sailing, andthe other objects may be objects not involved in sailing. Also, anobject involved in sailing may be an obstacle, and an object notinvolved in sailing may be the sky.

As another method for improving the accuracy of a neural network,different neural networks may be used according to conditions. Forexample, a neural network may vary according to a sea color.Alternatively, a neural network may vary according to zones. A neuralnetwork may be selected automatically or passively by a user. Forexample, a zone may be detected using a global positioning system (GPS),and then a neural network may be changed automatically according to thezone. Alternatively, a sea color may be detected by a sensor or thelike, and a neural network may be automatically changed.

To obtain a training image and/or an input image, a camera may beinstalled on a ship. The ship may be sailed by people or may be anautonomous ship. Alternatively, the ship may be neither a ship sailed bypeople nor an autonomous ship. Also, the camera may be installed in aharbor, a land, or the like rather than the ship without locationallimitations.

When a camera is installed in a ship, there is no limitation on thelocation at which the camera is installed. The camera may be installedat any location, such as a front side, a lateral side, or a rear side ofthe ship. Also, the camera may be installed toward the front of the shipor a lateral side or the rear.

A sensing range may vary according to a camera. For example, a distanceand/or an angle (field of view (FOV)) sensible by the camera may varyaccording to a lens and/or a focal length of the camera. The sensingrange of a camera may vary according to the size, purpose, etc. of aship. For example, in a large ship, a camera having a longer sensingrange than a camera in a small ship may be installed.

A single camera may be installed on a ship. Alternatively, a pluralityof cameras may be installed. When a plurality of cameras are used, thesensing range may be increased compared to that of the case in which asingle camera is used. For example, the FOV may widen, or detectedregions may be increased. Alternatively, when a plurality of cameras areused, the accuracy of situation awareness may be improved.

The plurality of cameras may be installed at difference locations. Forexample, a first camera may be installed on the front side of the ship,and a second camera may be installed on the rear side of the ship.Alternatively, the plurality of cameras may be installed at differentheights. For example, a second camera may be installed above a firstcamera.

The plurality of cameras may have different sensing regions. For thisreason, the plurality of cameras may be more effective in situationawareness compared to a single camera. As an example, a first camera maysense a region around the ship, and a second camera may sense a regionfar from the ship. Specifically, when a first camera equipped with awide angle lens to sense an object at a short distance (e.g., 0 m to 150m) and a second camera equipped with a zoom lens to sense an object at along distance (e.g., 150 m to 300 m) are used together, it is possibleto sense objects at both of the short distance and the long distance(e.g., 0 m to 300 m). As another example, a first camera may sense aregion in front of the ship, a second camera may sense a region on alateral side of the ship, and a third camera may sense a region behindthe ship. As another example, a first camera may sense a left frontregion of the ship, and a second camera may sense a right front regionof the ship. The sensing ranges of the plurality of cameras maypartially overlap.

When the plurality of cameras is used, images generated by each of theplurality of cameras may be stitched into one image and then input to aneural network such that image segmentation may be performed.Alternatively, some images generated by each of the plurality of camerasmay be stitched, and then image segmentation may be performed.

Alternatively, images generated by each of the plurality of cameras maybe separately input to a neural network such that image segmentation maybe performed. For example, assuming that distance information ispresented as short distance, middle distance, and long distance throughimage segmentation, when image segmentation is performed using a firstimage generated by a first camera for sensing an object at a distance of0 m to 150 m and a second image generated by a second camera for sensingan object at a distance of 150 m to 300 m, short distance (e.g., 0 m to50 m), middle distance (e.g., 50 m to 100 m), and long distance (e.g.,100 m to 150 m) results are obtained from the first image, and shortdistance (e.g., 150 m to 200 m), middle distance (e.g., 200 m to 250 m),and long distance (e.g., 250 m to 300 m) results are obtained from thesecond image, such that image segmentation results in which a total ofsix stages of distance ranges are classified may be obtained.

The number of cameras installed may vary according to the size, purpose,etc. of the ship. For example, the number of cameras installed in alarge ship may be greater than the number of cameras installed in asmall ship.

When a camera is installed in a ship, initial setting may be requiredfor various reasons, such as the type of ship and the location at whichthe camera is installed. Initial setting may be to set the camera sothat image segmentation results based on pixels shown in a camera videoand/or image or a neural network may become normal.

According to an initial setting method, after an appropriateinstallation guide may be given to a user, result values of an AISand/or a radar and the camera may be compared such that a module mayfind the location and orientation of the camera.

Alternatively, after a camera is installed first, a setting value of thecamera may be calculated to minimize an error according to results oftest sailing, and then the setting value may be used for situationawareness. In particular, the setting value may be calculated tominimize the error on the basis of the correlation between seamarks orthe like of an electronic navigational chart and result values of imagesegmentation. After that, when the calculated result value does not showenough distance because, for example, the camera is installed too fardownward, a warning may be given, and reinstallation may be recommended.

A processing method for the case of using an image captured by a cameraas an input image for image segmentation will be described below.

A camera may capture images at regular time intervals. The timeintervals may be fixed or dynamically changed.

An image acquisition period of a camera and an image segmentationoperation period may differ from each other. For example, imagesegmentation may be performed for only some images obtained through thecamera. When the camera obtains images of 60 frames per second, imagesegmentation may be performed on the basis of images of 10 frames amongthe 60 frames. According to a method of selecting some images, imagesmay be randomly selected. Otherwise, clear images may be selected fromthe captured images. Otherwise, images of a plurality of frames may becombined into one image, and then image segmentation may be performed.For example, an image obtained by combining a high illumination imageand a low illumination image may be used for segmentation.

One image may be used to perform image segmentation once, or a pluralityof images may be used. The plurality of images may be images captured bythe same camera or different cameras.

When image segmentation is performed using images captured by differentcameras, sensing ranges of the cameras may differ from each other. Forexample, a first camera may capture a short-distance image, and a secondcamera may capture a long-distance image.

Surroundings may be sensed in a way other than image segmentation. Forexample, it is possible to sense surroundings using an obstacledetection sensor, such as a radar, a LiDAR, and an ultrasonic detector,or information obtained with the obstacle detection sensor may beprocessed through a neural network to sense surroundings. Hereinafter,location information obtained as results of image segmentation will bereferred to as first location information, and location informationobtained in a way other than image segmentation will be referred to assecond location information.

In the case of an image captured by a camera, it is possible to obtainlocation information of each pixel of the image using settinginformation including the type, location (height, angle, etc.), etc. ofthe camera. The obtained location information may be an absolutelocation or a location relative to the location of the camera or alocation determined on an arbitrary basis.

FIG. 11 is a diagram relating to location information obtainment basedon image pixels. Referring to FIG. 11, the left-right direction in theimage may correspond to the angle of a pixel, and the up-down directionmay correspond to the distance to a pixel. A pixel A may be consideredto be about 15 m away at about 30° to the left, and a pixel B may beconsidered to be about 20 m away at about 10° to the right.

An error may occur in the above-described location information accordingto the position of a camera. For example, an initial installationposition of the camera may be changed, and an error may occuraccordingly. It is possible to correct the location information on thebasis of the installation position.

An error may occur in location information according to the movement(position) of an object in which a camera is installed. For example,when a camera is installed on a ship, location information of each pixelin an image may vary according to roll, pitch, etc. of the ship. Toobtain accurate location information, the position of an object in whicha camera is installed, such as the position of a ship, may be taken intoconsideration. For example, position information, such as heading oryaw, pitch, and roll, of a ship may be obtained from an inertialmeasurement unit (IMU) installed in the ship, and location informationof each pixel in an image may be obtained on the basis of the positioninformation. FIG. 12 is a set of diagrams relating to locationinformation obtainment based on the position of a ship according to anexemplary embodiment of the present invention. In FIG. 12, only thepitch of the ship is taken into consideration. When comparing the casesin which the bow of a ship is directed upward and downward, it ispossible to see that a distance corresponding to the same location onimages may vary. Referring to FIG. 12, both pixels A and B are at thecenter of images. However, the distance of the pixel A is 40 m, and thedistance of the pixel B is 10 m, that is, there is a difference indistance. Therefore, unless the pitch of the ship is taken intoconsideration through an inertia measurement device, distanceinformation may be inaccurate.

It is possible to sense surroundings by combing image segmentation and amethod other than image segmentation. Alternatively, informationobtained through image segmentation may be corrected in a way other thanimage segmentation. Here, correction is not limited to obtainingabsolutely accurate information and also denotes that information ischanged.

Fusion of first location information and second location informationwill be described in detail below.

FIG. 13 is a block diagram relating to acquisition of final locationinformation according to an exemplary embodiment of the presentinvention. Referring to FIG. 13, first location information may beobtained through image segmentation S1500, second location informationmay be obtained in a way other than image segmentation, and then finallocation information may be obtained by considering first locationinformation and second location information.

It is unnecessary to fuse location information corresponding to alloutput values of output data of image segmentation together. Finallocation information of some output values may be obtained using thefirst location information and the second location information, andfinal location information of other output values may be obtained usingonly one of the first location information and the second locationinformation. For example, it is possible to determine whether to fusepieces of location information according to the type of object obtainedthrough image segmentation. First location information of an obstaclemay be fused with second location information, and that of a sailableregion, such as the sea, may not be fused with second locationinformation.

The type of first location information may differ from the type ofsecond location information. For example, the first location informationmay contain only distance information, and the second locationinformation may contain only direction information. In this case, finallocation information may be obtained by combining the first locationinformation and the second location information.

The first location information and the second location information mayinclude the same type of information. In this case, the first locationinformation and the second location information may or may notcorrespond to each other.

When the pieces of location information correspond to each other, thefirst location information and the second location information may beidentical or similar to each other or within the margin of error. Forexample, when the pieces of location information correspond to eachother, the first location information may be presented as a distanceregion, and the second location information may be included in thedistance region of the first information. Final location information maybe one of the first location information and the second locationinformation. Alternatively, final location information may be a valueobtained by correcting the first location information with the secondlocation information.

FIG. 14 is a diagram relating to acquisition of final locationinformation according to an exemplary embodiment of the presentinvention when pieces of location information correspond to each other.When first location information is presented as a distance region, anoutput value of −1 indicates a distance of less than 10 m, an outputvalue of 0 indicates a distance of 10 m or more and 20 m or less, anoutput value of 1 indicates a distance of more than 20 m, and secondlocation information is presented as a distance value in units ofmeters. Final location information is presented as a distance region onthe basis of the first location information and the second locationinformation. As an example, the first location information may determinethe maximum value of final location information, and the second locationinformation may determine the minimum value of final locationinformation. An output value of the first location information at thecenter is 0, and that of the second location information at the centeris 14 m. The maximum is determined to be 20 m by the first locationinformation, and the minimum is determined to be 14 m by the secondlocation information such that final location information may be adistance range of 14 m to 20 m.

As another example, both of the first location information and thesecond location information may be presented as distance values andcorrespond to each other. In this case, final location information maybe any one of the first location information and the second locationinformation. Alternatively, final location information may be theaverage of the first location information and the second locationinformation. In addition to this, various methods may be used, such asmixing the first location information and the second locationinformation at a specific ratio.

When pieces of location information do not correspond to each other, thefirst location information and the second location information aredifferent or fall outside the margin of error. In this case, finallocation information may be one of the first location information andthe second location information. For example, final location informationmay be a smaller value between the first location information and thesecond location information. Otherwise, pieces of location informationof an output value which do not correspond to each other may be left asunknown values. Otherwise, when pieces of location information of anoutput value do not correspond to each other, location information ofall output values of the corresponding output data may be left asunknown values.

FIG. 15 is a diagram relating to acquisition of final locationinformation according to an exemplary embodiment of the presentinvention when pieces of location information do not correspond to eachother. FIG. 15 shows first location information and second locationinformation in the same way as FIG. 14. Referring to FIG. 15,highlighted output values at the top-left corner do not correspond toeach other, and the first location information takes precedence and isselected as final location information.

According to an exemplary embodiment, pieces of location information maybe fused together for each object. For example, first locationinformation and second location information may correspond to each of aplurality of pixels corresponding to a specific object in an image. Inthis case, final location information may be acquired regarding theobject rather than each pixel by fusing first location information andsecond location information together.

According to an exemplary embodiment, fusing pieces of locationinformation for each object may include calculating first objectlocation information on the basis of a plurality of pieces of firstlocation information with which a plurality of pixels corresponding to aspecific object are separately labeled, calculating second objectlocation information on the basis of a plurality of pieces of secondlocation information with which the plurality of pixels corresponding tothe specific object are separately labeled, and fusing the first objectlocation information and the second object location information toacquire final location information corresponding to the object. As anon-limiting example, the object location information may be calculatedthrough the average, the weighted average, the outlier processing, andthe like of a plurality of pieces of location information. Also, theabove-described fusing method for pieces of location information whichmatch each other and the above-described fusing method for pieces oflocation information which mismatch each other may be applied to adetailed method of fusing the first object location information and thesecond object location information together.

When image segmentation is performed, second location information may beused as input data together with an image. FIG. 16 is a diagram relatingto acquisition of final location information according to an exemplaryembodiment of the present invention. Referring to FIG. 16, an image andsecond location information may be used as inputs to image segmentationS1500 to obtain final location information. When the image segmentationS1500 is performed by a neural network, the second location informationmay be input in the same operation as the image or in a differentoperation from the image. For example, when the neural network includesa plurality of sequential layers, the image may be input to the firstlayer, and the second location information may be input to anotherlayer. As a result, calculation may be performed with the image in allthe layers and may not be performed with the second location informationin one or more layers.

The case in which one piece of first location information and one pieceof second location information are used to obtain one piece of finallocation information has been described above. However, the presentinvention is not limited thereto, and a plurality of pieces of firstlocation information and/or a plurality of pieces of second locationinformation may be used to obtain one piece of final locationinformation. Also, it has been assumed above that second locationinformation is presented as an actual distance value, but the presentinvention is not limited thereto.

The path-planning operation may be an operation of receiving objectinformation and outputting a path followed by a ship (hereinafter,referred to as a “following path”). FIG. 17 is a block diagram relatingto the path-planning operation according to an exemplary embodiment ofthe present invention. Referring to FIG. 17, a starting point and adestination of a ship, ship status information, weather environmentinformation, sailing regulations, etc. may be received in addition toobject information, and a following path may be output through thepath-planning operation S3000.

The starting point and the destination may be an initial startinglocation and a final arrival location of the ship but are not limitedthereto. For example, the starting point and the destination may bearbitrary marine locations. Otherwise, the starting point may be acurrent location of the ship, and the destination may be the finalarrival location. Otherwise, the starting point may be a currentlocation of the ship, and the destination may be an arbitrary locationon an obstacle map.

The ship status information refers to information relating to the ship,such as the location, speed, and position information of the ship. Theposition information refers to information relating to the position ofthe ship, such as heading, pitch, and roll, of the ship.

The weather environment information refers to information related to aweather environment, such as tidal current, wind, and waves.

The sailing regulations refer to agreements, regulations, or the likethat are enforced for ship sailing. For example, the sailing regulationsmay be the international regulations for preventing collisions at sea(COLREG).

The following path is a path followed by the ship and may be a waypointwhich represents the location passed by the ship. Alternatively, thefollowing path may be a path along which the ship moves.

It is possible to generate the following path on the basis ofinformation on seamarks. The seamarks may be obtained by situationawareness, such as image segmentation.

Input information, for example, information received in thepath-planning operation, may include all the pieces of above-describedinformation, only some thereof, or other information.

The path-planning operation may be implemented through one algorithm orneural network. For example, it is possible to train and use a neuralnetwork which outputs a following path when object information is input.Alternatively, a plurality of algorithms or neural networks may be used.

The path-planning operation may include an obstacle map update operationand a path generation operation. FIG. 18 is a block diagram relating tothe path-planning operation according to an exemplary embodiment of thepresent invention. In an obstacle map update operation S3500, objectinformation may be received, and an obstacle map may be output. In apath-generation operation S3700, the obstacle map may be received, and afollowing path may be output.

FIG. 19 is a block diagram relating to an obstacle map update operationaccording to an exemplary embodiment of the present invention. Referringto FIG. 19, it is possible to update an obstacle map by consideringobject information, weather environment information, such as tidalcurrent and wind, and sailing regulations.

The object information is not limited to object information obtainedthrough image segmentation, and object information obtained throughanother sensor, such as a radar or a LiDAR, may also be an input to theobstacle map update operation. It is also possible to combine all orsome of the pieces of object information.

The obstacle map refers to a means for presenting object information. Asan example, the obstacle map may be a grid map. In the grid map, thespace may be divided into unit regions, and object information may bedisplayed according to each unit region. As another example, theobstacle map may be a vector map. The obstacle map is not limited tobeing two dimensional and may be a 3D obstacle map. Meanwhile, theobstacle map may be a global map which presents all zones related toship sailing from a starting point to a destination or a local map whichpresents certain zones around the ship.

The obstacle map may include a plurality of unit regions. The pluralityof unit regions may be presented in various ways according toclassification criteria. As an example, the obstacle map may include asailable region, an unsailable region, and an unknown region which maynot be determined to be sailable or not. Specifically, the sailableregion may be a region in which there is no obstacle or any obstacle ishighly unlikely to exist, and the unsailable region may be a region inwhich there is an obstacle or an obstacle is highly likely to exist.Alternatively, the sailable region may correspond to a suitable zone forsailing, and the unsailable region may correspond to an unsuitable zonefor sailing. The unknown region may be a region excluding the sailableregion and the unsailable region.

As another example, the obstacle map may include a region correspondingto a zone within the sensing range of a camera, an obstacle sensor, orthe like and a region not corresponding thereto. The sensing range of acamera may be presented as an FOV. The region corresponding to the zonewithin the sensing range may include a detected region and a undetectedregion which is a region other than the detected region. Unit regionsmay be classified as detected regions and undetected regions accordingto whether it is possible to obtain object information through a camera,an obstacle detection sensor, or the like. Alternatively, unit regionsmay be classified as detected regions and undetected regions accordingto the accuracy of object information obtained through a camera, anobstacle detection sensor, or the like. For example, a zone occluded byan obstacle may correspond to an undetected region.

Otherwise, the obstacle map may include an update region and anotherregion according to whether the corresponding region has been updated.

A weight may be allocated to each unit region of the obstacle map. Theweight may reflect sailing suitability. For example, a weight of a highvalue may be allocated to a unit region corresponding to a suitable zonefor sailing, such as the sea, and a weight of a low value may beallocated to a unit region corresponding to a zone which is unsuitablefor sailing because there is an obstacle such as a ship. In reverse, asmaller weight may be allocated to a unit region having higher sailingsuitability.

Presentation of an obstacle on a 2D obstacle map will be mainlydescribed below, but the present invention is not limited thereto.

The term “update” may be construed as including generation. For example,when there is no obstacle map of a previous frame in the case ofupdating an obstacle map of a current frame, updating an obstacle mapmay be construed as generating an obstacle map.

On the obstacle map, object information may be presented. For example,it is possible to show whether an object exists. Also, it is possible topresent various information, such as the type of object, the location ofan object, the movement direction of an object, the speed of an object,and navigation mark information. On the obstacle map, only informationon obstacles may be shown. The information may be presented as whetherthere is an obstacle or the probability of presence of an obstacle.Alternatively, the information may be presented as a weight using anumber in a certain range.

When obstacles are presented as weights, the weights may be categorizedinto a plurality of numeric ranges. The respective numeric ranges maycorrespond to the sailable region, the unsailable region, and theunknown region. FIG. 20 is a set of diagrams relating to an obstacle mapaccording to an exemplary embodiment of the present invention. FIG. 20shows examples of a 2D grid map. Referring to (a) of FIG. 20, whetherthere is an obstacle in each grid and/or unit region or whether eachgrid and/or unit region is suitable for sailing is presented using aninteger of 0 to 255 as a weight. A smaller value represents that theprobability of presence of an obstacle is high or the unit region isunsuitable for sailing, and a unit region without a value has a weightof 255. Unsailable regions may correspond to a weight of 0 to a, unknownregions may correspond to a weight of a+1 to b, and sailable regions maycorrespond to a weight of b+1 to 255. a and b are integers satisfying0≤a<b≤254. For example, unsailable regions may correspond to a weight of0, sailable regions may correspond to a weight of 255, and unknownregions may correspond to a weight of 1 to 254. Alternatively,unsailable regions may correspond to a weight of 0 to 50, sailableregions may correspond to a weight of 51 to 200, and unknown regions maycorrespond to a weight of 201 to 255. Referring to (b) of FIG. 20, theobstacle map presented with weights may be visualized. For example, theobstacle map may be presented by adjusting the brightness of anachromatic color. In (b) of FIG. 20, a smaller weight is presented withlower brightness. A detailed method of allocating a weight to a unitregion will be described below.

Object information for updating an obstacle map may be information basedon an attribute of a means for obtaining the object information. Forexample, location information of an obstacle obtained through imagesegmentation may be location information relative to a camera which hascaptured an image input for the image segmentation. The relativelocation information may be converted into location information on theobstacle map so as to be applied to the obstacle map. One or both of therelative location information and the location information on theobstacle map may be used to update the obstacle map. Hereinafter,coordinates on an obstacle map are referred to as absolute coordinates.

It is possible to obtain absolute information using a GPS, an IMU,direction information of a ship and/or camera, or the like. FIG. 21 is ablock diagram relating to a location information conversion operationaccording to an exemplary embodiment of the present invention. Referringto FIG. 21, location information of an obstacle may be converted intoabsolute coordinates on the basis of the location of a ship and/or acamera calculated with a GPS installed in the ship, position informationof the ship obtained by an IMU, and camera position information based oncamera installation information.

In the case of an obstacle map update, the entire obstacle map may beupdated. Alternatively, only some regions of the obstacle map may beupdated.

An update region may be defined on the basis of an update distance andan update angle. FIG. 22 is a set of diagrams relating to an updateregion according to an exemplary embodiment of the present invention andshows a global map 310 including a starting point A, a destination B,and a current location C of a ship and a local map 330. An update region331 may be defined in the local map 330 with an update distance r and anupdate angle θ.

An update region may be determined according to the location of a ship,heading of the ship, and the like. Alternatively, an update region maybe determined according to an obstacle detection sensor, such as acamera, a radar, or a LiDAR. For example, an update region may bedetermined by the angle of view and/or the maximum observation distanceof a camera. An update angle may be determined by the angle of view of acamera, and an update distance may be determined by the maximumobservation distance. Alternatively, an update region may be a certainregion around a ship. An update region may be larger than a minimumregion for detecting and then avoiding an obstacle.

The case of presenting whether there is an obstacle in an obstacle mapand/or whether a unit region is suitable for sailing with a weight willbe described in detail below. It is assumed below that a small weightrepresents a high probability of presence of an obstacle orunsuitability for sailing, but the present invention is not limitedthereto.

A weight may vary according to the type information of an object. As anexample, when a ship and a buoy are detected as obstacles, the weight ofthe ship may be set to be smaller than the weight of the buoy. Asanother example, when a stationary object and a moving object aredetected, the weight of the stationary object may be set to be smallerthan the weight of the moving object.

A weight may be set on the basis of location information of an objectwhich is corrected by considering type information of an object.Alternatively, a weight may be set for a region in which an object hasnot been detected on the basis of type information of the object.

FIG. 23 is a block diagram relating to correction of locationinformation according to an exemplary embodiment of the presentinvention. Referring to FIG. 23, type information and locationinformation of an object included in an image may be obtained throughimage segmentation, and an obstacle map may be updated after theobtained type information applied to the obtained location information.

As an example, the size of an object may be calculated on the basis oftype information of the object, and unit regions corresponding to theobject on an obstacle map may be calculated using the size such thatweights may be set. The unit regions may vary according to a directionand the like of the object.

As another example, distance information obtained through imagesegmentation may include only the minimum distance to an object. Whentype information of the object is additionally used, it is possible toobtain the maximum distance as well as the minimum distance, and weightsmay be set on the basis of the distance information.

FIG. 24 is a set of diagrams relating to weight setting according to anexemplary embodiment of the present invention. The image of (a) of FIG.24 may be input, and distance information may be obtained through imagesegmentation. The distance information may include only the minimumdistance to an object. In this case, referring to (b) of FIG. 24, anundetected region 351, in which it is not possible to know whether anobstacle exists, may be in the image. On an obstacle map, the undetectedregion 351 may be processed as a region in which an obstacle exists or aregion in which no obstacle exists. Alternatively, a part of theundetected region 351 may be processed as a region in which an obstacleexists, and another part may be processed as a region in which noobstacle exists.

It is possible to reduce unit regions corresponding to the region 351,in which it is not possible to know whether an obstacle exists, on theobstacle map using type information of the obstacle. For example, asshown in (c) of FIG. 24, the size of the obstacle may be calculated fromthe type information of the obstacle, and a region corresponding to thesize on the obstacle map may be processed as an unsailable region.Alternatively, certain unit regions may be added to the regioncorresponding to the size of the obstacle on the obstacle map, and theresultant region may be processed as an unsailable region. A weight ofthe certain unit regions may differ from a weight of the regioncorresponding to the size of the obstacle on the obstacle map. The sizeof the obstacle may be an accurate size of the obstacle or may bedetermined on the basis of the type of obstacle. Alternatively, the sizeof the obstacle may be a value previously determined according to thetype of obstacle.

Referring to (d) of FIG. 24, when the size of the obstacle is obtainedfrom the type information of the obstacle, there may be a zone in whichit is not possible to know whether an obstacle exists. The zone may bean occluded zone which is covered by the obstacle and thus is not shown.Location information of the occluded zone may be inaccurate information.A weight of unit regions which correspond to the occluded zone on theobstacle map may be a value between the weight of unit regions in whichthe obstacle exists and the weight of unit regions in which no obstacleexists.

The obstacle map may be updated by calculating a weight from correctedlocation information. Alternatively, a weight may be calculated from thelocation information before correction to update the obstacle map, andthen the weight may be corrected by considering the type information.

A weight may be set by considering a movement direction and speed of anobject. For example, a small weight may be set for a direction in whichthe object moves. Alternatively, a smaller weight may be set for highermovement speed. FIG. 25 is a set of diagrams relating to weight settingin which movement of an obstacle is taken into consideration accordingto an exemplary embodiment of the present invention. In the diagrams ofFIG. 25, weights are presented by considering a movement direction andspeed according to the object information presentation method of FIG.20. Referring to (a) of FIG. 25, when a ship which is an obstacle isstopped, an obstacle map may be divided into an unsailable region 353corresponding to the ship and a sailable region 355 corresponding to thesea. When the ship moves leftward, referring to (b) of FIG. 25, a weightof a predicted movement region 357 on the left side of the ship in theobstacle map may be set to a value between a weight of the unsailableregion 353 and a weight of the sailable region 355. Referring to (c) ofFIG. 25, as the movement speed of the ship increases, the weight of thepredicted movement region 357 may be reduced. Alternatively, as themovement speed of the ship increases, the area of the predicted movementregion 357 may be increased.

The danger of an object may be determined so that a weight may be setdifferently. For example, a smaller weight may be set for an object withhigher danger. The danger may be calculated by considering objectinformation, such as the type, movement direction, and speed of anobject.

A weight may be set by considering weather environment information suchas tidal current and wind. For example, a small weight may be set forunit regions to which it is difficult to move because the unit regionsare present in a direction in which the ship moves against a tidalcurrent, wind, or the like, and a large weight may be set for unitregions to which it is easy to move because the unit regions are presentin a direction in which the ship moves with a tidal current, wind, orthe like. Data of a tidal current, wind, etc. may be obtained through anadditional sensor. Otherwise, tidal current and/or wind informationdependent on a zone or the time may be stored in advance and used.Otherwise, the information may be received in real time from an externalsource.

Geographical features may be presented on an obstacle map through thesituation awareness operation or may be presented on an obstacle mapwith reference to an electronic navigational chart or the like withoutsituation awareness. Electronic navigational chart information andsituation awareness results may be used together. Geographical featuresmay be set as an unsailable region on an obstacle map.

A weight may be set by considering sailing regulations such as COLREG.

According to sailing regulations, it is possible to increase a weight ofa region in which a ship will sail or reduce a weight of a region inwhich a ship should not sail. For example, according to COLREG, when twoships move toward each other, one ship may be required to detour to theleft side of a movement direction. In this case, a weight of unitregions on the right side may be increased, or a weight of unit regionson the left side may be reduced so that a ship may sail while satisfyingCOLREG. FIG. 26 is a set of diagrams relating to weight setting in whichsailing regulations are taken into consideration according to anexemplary embodiment of the present invention. When COLREG is not takeninto consideration, only unit regions corresponding to a ship movingtoward a corresponding ship may be shown as an unsailable region 353 onan obstacle map. A weight of sailing regulation reflecting regions 359which are left unit regions on the obstacle map may be reduced so thatCOLREG may be satisfied. The weight of the sailing regulation reflectingregions 359 may be set to a value between a weight of a sailable region355 and a weight of the unsailable region 353. Due to the sailingregulation reflecting regions 359, a path for detouring to the rightside of the ship moving toward the corresponding ship may be generated.

A buffer region having an arbitrary weight may be provided aroundunsailable regions and/or sailable regions. The weight of the bufferregion may be set to a value between a weight of sailable regions and aweight of unsailable regions. Alternatively, the weight of the bufferregion may be set by considering the distance from an unsailable regionand/or a sailable region. For example, a region close to a sailableregion may have a larger weight than a region close to an unsailableregion. FIG. 27 is a diagram relating to weight setting in which abuffer region is taken into consideration according to an exemplaryembodiment of the present invention. In the diagram of FIG. 27, a localmap 330 is presented according to the object information presentationmethod of FIG. 20, and an unsailable region 353, a sailable region 355,and a buffer region 358 are shown in an update region 331.

An update region of an obstacle map may be updated with objectinformation.

The obstacle map may be updated with information about whether anobstacle exists. For example, when an obstacle is detected, the obstaclemay be displayed. Alternatively, a weight may be updated in the obstaclemap. For example, a weight may be calculated using object information,and the obstacle map may be updated with the weight.

An obstacle map update according to the object information presentationmethod of FIG. 20 will be described below. However, an obstacle mapupdate method is not limited thereto and may also be applied to the casein which weight ranges are different, the case in which an obstacle mapis presented with the probability of presence of an obstacle, and othercases.

Object information of a previous frame may be ignored, and an obstaclemap may be updated only with object information obtained from a currentframe. For example, an obstacle map may be generated only withinformation obtained through the above-described weight setting.

Alternatively, an obstacle map may be updated by considering objectinformation of both a previous frame and a current frame. For example,an obstacle map may be updated by considering a weight of a currentframe and a weight of a previous frame or an obstacle map obtainedthrough the above-described weight setting.

FIG. 28 is a block diagram relating to an obstacle map update operationaccording to an exemplary embodiment of the present invention. Referringto FIG. 28, it is possible to generate an obstacle map of a currentoperation by considering an obstacle map of the previous operation(S3500). For example, a final obstacle map of a current frame may begenerated by adding the obstacle map of the previous frame and theobstacle map of the current frame at a certain ratio. A plurality ofprevious frames may be used to generate a final obstacle map.

The obstacle map may be updated by ignoring the previous frame regardinga region detected in a marine image of the current frame and consideringthe previous frame regarding a region not detected in the marine imageof the current frame. For example, a weight of unit regions determinedto be an obstacle in the current frame may be set to 0, a weight ofsailable regions may be set to 255, and a weight of undetected regionsmay be set by considering a weight of the previous frame and a weight ofthe current frame at a certain ratio. The weight of undetected regionsmay converge into a specific value over time. Also, the specific valuemay vary according to types of objects. For example, a weight of astationary obstacle may converge into a smaller value than that of amoving obstacle. Alternatively, the weight of undetected regions may notbe updated.

Unit regions corresponding to a zone which is not within the FOV of themarine image of the current frame may be processed in the same way asdescribed above regarding the undetected regions.

A weight of specific regions may not be updated. For example, when astationary obstacle which does not move is detected in a region, aweight of the region may be fixed at a specific value, such as 0, andmay not change over time. The stationary obstacle may be a land, anisland, a rock, or the like.

FIG. 29 is a block diagram relating to an obstacle map update operationaccording to an exemplary embodiment of the present invention. Referringto FIG. 29, it is possible to convert location information of anobstacle into absolute coordinates using the location of the ship and/orthe camera calculated with the GPS installed in the ship, the positioninformation of the ship obtained by the IMU, and the camera positioninformation based on the camera position information (S3100). A weightmay be set using the converted object information, weather environmentinformation, and sailing regulations (S3300), and a final obstacle mapof a current frame may be output (S3500) using the obstacle map of theprevious frame and an update region set on the obstacle map (S3600).

In a path generation operation, a path which is sailable by the ship isgenerated. It is possible to generate the path by considering a movementdistance, a movement time, the costs of sailing, and the like of theship. The path may be generated by additionally considering thesurroundings of the ship, such as tidal current and wind, and sailingregulations, such as COLREG, may also be taken into consideration.

Different paths may be generated according to circumstances byconsidering the minimum distance, the minimum time required, the minimumcost, the danger, and the like. For example, since the probability thata person is present is higher in the neritic zone than in the oceaniczone, the path may be generated in a high safety mode for safe sailing.In the oceanic zone or in the case of a large ship, the path may begenerated in a high energy efficiency mode to reduce fuel consumption.The path may be generated in an optimal manner by considering aplurality of variables at an appropriate ratio. Such a mode may beselected manually or automatically. As an example, the path in theneritic zone may be generated in the high safety mode, which isautomatically switched to the high energy efficiency mode using shiplocation information of a GPS or the like, and the path in the oceaniczone may be generated in the high energy efficiency mode. As anotherexample, when many obstacles are detected, the mode may be automaticallyswitched to the high safety mode to generate the path.

FIG. 30 is a block diagram relating to a path generation operationaccording to an exemplary embodiment of the present invention. Referringto FIG. 30, a following path may be generated using the obstacle map,the starting point, the destination, the ship status information, theweather environment information, and the sailing regulations as inputs(S3700). Input information may include all or only some of theaforementioned information and may also include information other thanthe aforementioned information.

The following path may be generated using a neural network. For example,a neural network which outputs a following path using an obstacle map orthe like as an input may be trained and used in the path generationoperation.

An algorithm other than a neural network may be used to generate thefollowing path. Any path generation algorithm, such as Dijkstra'salgorithm, A* algorithm, D* algorithm, and Theta* algorithm, may be usedwithout limitations.

The path may be generated by considering heading, the maximum rudderangle of the ship, and the like. FIG. 31 is a diagram relating to a pathgeneration operation according to an exemplary embodiment of the presentinvention. Referring to FIG. 31, a ship 371 is facing a specific heading373. The path of a solid line may be generated when the heading 373 ofthe ship 371 and the maximum rudder angle are not taken intoconsideration, and the path of a broken line may be generated when theheading 373 of the ship 371 and the maximum rudder angle are taken intoconsideration. The path may be presented with waypoints 375. The path ofthe solid line may be impossible or difficult for the ship to follow. Onthe contrary, the path of the broken line may be easy or possible forthe ship to follow. Such an algorithm specialized for ships may be usedto generate a path useful for path following.

In the path-following operation, a control signal is generated so thatthe ship may follow the planned path. FIG. 32 is a block diagramrelating to a path-following operation according to an exemplaryembodiment of the present invention. Referring to FIG. 32, in thepath-following operation S5000, a following path, ship statusinformation, weather environment information, sailing regulations, etc.may be input, and a control signal for the ship may be output. The inputinformation may include all or only some of the aforementionedinformation and may also include information other than theaforementioned information.

The control signal includes a speed control signal and a heading controlsignal. The speed control signal may be a signal for adjusting therotational speed of a propeller. Alternatively, the speed control signalmay be a signal for adjusting revolution per unit time of the propeller.The heading control signal may be a signal for adjusting the rudder.Alternatively, the heading control signal may be a signal for adjustingthe wheel or helm.

Also, a control signal of a previous frame may be used in thepath-following operation of a current frame.

A neural network may be used to perform the path-following operation. Inthis case, any method, such as supervised learning, unsupervisedlearning, reinforcement learning, and imitation learning, may be used totrain the neural network without limitations.

When reinforcement learning is used, the neural network may be trainedby providing rewards according to a sailing time, a sailing distance,safety, energy efficiency, and the like. For example, when the sailingtime is short, a high reward may be provided. As the sailing distancebecomes shorter, a higher reward may be provided. As the number ofcollisions with obstacles becomes smaller, a higher reward may beprovided. As the amount of fuel used becomes small, a higher reward maybe provided. Rewards may be determined in various other ways as well,and rewards may be provided by considering a plurality of variablesrather than only one variable. For example, it is possible to select areward setting method in which both of the sailing time and the energyefficiency are taken into consideration.

The path-following operation may be performed using anotherpath-following algorithm. For example, it is possible to use algorithms,such as model predictive control (MPC), which is a technology forreceiving locations to travel to for an arbitrary time period T from acurrent point in time and generating an optimal control signal, purepursuit, the Stanley method, and vector pursuit.

Even in autonomous navigation, information may be required for a personto monitor the ship. Such information may be transferred to a personthrough visualization. Results used or generated in the above-describedsituation awareness, path-planning, and path-following operations may beprocessed and visualized.

Visualization data may be generated from one image. Alternatively,visualization data may be generated from a plurality of images. Theplurality of images may be images obtained from different cameras aswell as images obtained from the same camera.

When a plurality of images are used for visualization, regions presentedby the images may differ from each other. In this case, the plurality ofimages may be stitched for visualization. For example, a plurality ofimages presenting surrounding regions of the ship may be stitched suchthat around-view monitoring may be possible. Around-view monitoring mayprovide surroundings of the ship in a bird's eye view.

When image segmentation results are visualized, results showing anobject detected through the image segmentation may be output through adisplay panel. For example, the sea, a ship, geographical features, anavigation mark, etc. included in an image may be presented in differentcolors.

Also, it is possible to output obstacle characteristics including thedistance, speed, danger, size, and collision probability of an obstacle.Obstacle characteristics may be output using color, which may varyaccording to the distance, speed, danger, size, and collisionprobability of an obstacle.

Obstacle characteristics sensed by an obstacle sensor may be outputtogether. For example, when a region is observed through imagesegmentation, image segmentation results of the region may be output,and when a region is not observed, sensing results of the regionobtained by an obstacle sensor may be output.

The obstacle map, the following path, a path history, and/or the likemay be visualized. For example, a ship-oriented obstacle map, anexisting path, and an obstacle-avoiding path may be output in a bird'seye view.

FIG. 33 is a diagram relating to visualization according to an exemplaryembodiment of the present invention. Referring to FIG. 33, a displaypanel may output a relative position of an obstacle based on a currentlocation of a ship, characteristics of the obstacle, and the like. Forexample, a black region 610 may present a first obstacle which is closerthan a predetermined distance, and a grey region 630 may present asecond obstacle which is farther than the predetermined distance.Alternatively, the black region 610 may present a region with highdanger, the grey region 630 may present a region with medium danger, anda white region 650 may present a region with little danger.

FIG. 34 is a block diagram relating to an autonomous navigation methodbased on image segmentation according to an exemplary embodiment of thepresent invention. Referring to FIG. 34, an image may be obtained bycapturing the image with a camera (S1100), a preprocessed image may beobtained through a preprocessing process for the image (S1300), firstobject information may be obtained by segmenting the preprocessed image(S1500), an obstacle map may be updated with the first objectinformation and second object information obtained through an obstaclesensor or the like (S3500), a following path may be generated using theobstacle map, a starting point, a destination, and ship statusinformation (S3700), and a control signal of a current frame may begenerated using the following path, the ship status information, and acontrol signal of a previous frame (S5000). Image segmentation (S1500)may be performed using a neural network, and other operations may beperformed using the above-described algorithm other than a neuralnetwork. Alternatively, image segmentation may be performed using aneural network (S1500), the path-planning operation (S3500 and S3700)may be performed using a non-neural network algorithm, and thepath-following operation (S5000) may be performed using the neuralnetwork.

After a specific operation is repeated a plurality of times in oneframe, the next operation may be performed. For example, imagesegmentation (S1500) may be performed several times, and then pathplanning (S3500 and S3700) and path following (S5000) may be performed.Alternatively, image segmentation (S1500) may be performed severaltimes, and then obstacle map update (S3500) may be performed.

It is unnecessary to perform an operation on a frame after alloperations are performed on the previous frame. For example, while thepath generation operation (S3700) is performed on one frame, imagesegmentation (S1500) may be performed on the next frame. Alternatively,the path-following operation (S5000) of one frame, the path generationoperation (S3700) of the next frame, and the image segmentationoperation (S1500) of the second next frame may be performed together.

In the exemplary embodiment of FIG. 34, the situation awareness,path-planning, and path-following operations are performed separately.However, at least two of the situation awareness, path-planning, andpath-following operations may be performed through one neural network.Described below are a case in which path-planning and path-following areperformed through one neural network and a case in which all ofsituation awareness, path-planning, and path-following are performedthrough one neural network, but the present invention is not limitedthereto.

FIG. 35 is a block diagram relating to autonomous navigation based onimage segmentation. Referring to FIG. 35, an image may be obtained bycapturing the image with a camera (S1100), a preprocessed image may beobtained through a preprocessing process for the image (S1300), firstobject information may be obtained by segmenting the preprocessed image(S1500), and a control signal of a current frame may be generated usingthe first object information, second object information obtained throughan obstacle sensor, a starting point, a destination, ship statusinformation, and a control signal of a previous frame (S7000).

According to the autonomous navigation method of FIG. 34, a first neuralnetwork for image segmentation, a second neural network for pathplanning, and a third neural network for path following may beseparately trained and then used for an inference operation. However, aneural network which performs all of image segmentation, path planning,and path following may be trained at once and then used. Such learningof a neural network is referred to as end-to-end learning.

FIG. 36 is a block diagram relating to autonomous navigation based onimage segmentation. Referring to FIG. 36, image segmentation, pathplanning, and path following may be performed using one neural network.For example, a neural network which receives information, such as animage, object information from an obstacle detection sensor, ship statusinformation (the location and speed of a ship obtained from a GPS, theheading of the ship from an IMU, etc.) a starting point, a destination,waypoints, tidal current, wind, sailing regulations, and a controlsignal and outputs a ship control signal may be trained and used forautonomous navigation. The aforementioned pieces of input informationare examples. All or only some of the aforementioned information may beused, and information other than the aforementioned information may alsobe used.

FIG. 37 is a block diagram relating to a learning method of a neuralnetwork which outputs a control signal according to an exemplaryembodiment of the present invention. This learning method may be usefulwhen it is difficult to obtain a control signal as training data. In aninference operation, a global neural network is used. A local neuralnetwork is used for learning of the global neural network. The localneural network may handle linearity, and the global neural network mayhandle nonlinearity. The local neural network may receive a path and animage and output a control signal. A path-predicting neural network mayreceive a path, an image, and a control signal and output a subsequentpath. It is possible to train the local neural network through errortraining based on the control signal output by the local neural networkand the subsequent path output by the path-predicting neural network.The global neural network may receive a path and an image to output acontrol signal and may be trained through a discriminator which uses thecontrol signal output by the local neural network and the control signaloutput by the global neural network. There may be a plurality of localneural networks and a plurality of path-predicting neural networks. Theglobal neural network may learn errors from several local neural networkand handle all inputs and outputs as a single network.

FIG. 38 is a block diagram relating to an output of a neural networkaccording to an exemplary embodiment of the present invention and showsonly an output of the neural network of FIG. 36. In comparison to FIG.36, for visualization, it is possible to obtain object information, anobstacle map, a following path, etc. from a neural network. Hereinafter,the pieces of information are referred to as intermediate outputinformation. The intermediate output information may be visualizedthrough the above-described visualization method.

A separate neural network may be used to obtain the intermediate outputinformation. The neural network of FIG. 38 (hereinafter, referred to as“autonomous navigation neural network”) may obtain intermediate outputinformation through a neural network for visualization (hereinafter,referred to as “visualization neural network”). For example, thevisualization neural network may receive information output from theautonomous navigation neural network and output object information, anobstacle map, a following path, and the like. Several pieces ofintermediate output information may be obtained by one neural network.Alternatively, different neural networks may be used for respectivepieces of intermediate output information.

The visualization neural network may be trained separately from theautonomous navigation neural network. Alternatively, the visualizationneural network may be trained together with the autonomous navigationneural network.

As described above, objects may include obstacles, such as geographicalfeatures, structures, vessels, buoys, and people, a region in which avessel may sail, such as the ocean, and a region which may not berelated to sailing of a vessel, such as the sky. In addition, objectsmay include a backlight, a crane, a rope, and sea clutter. Here, thebacklight may denote a phenomenon in which information on an objectdisappears in an image or the like generated by a device due to the sun,lighting, or the like, or a region of the image corresponding to thephenomenon (e.g., a region appears to be white in a camera image due todirect light). Here, the rope may include a mooring rope for mooring avessel. In addition, an object which is a vessel may include a largevessel, a small vessel, and a tug. In addition, an object may includethe upper surface and the side surface of a vessel. In this case,information on a region or point at which the upper surface and the sidesurface come into contact with each other may be acquired on the basisof the upper surface and the side surface of the vessel. Also, an objectmay include a side line of a vessel corresponding to a region or pointat which the upper surface and the side surface of the vessel come intocontact.

Some methods of representing location information and additionalinformation according to exemplary embodiments are described below. Forconvenience of description, location information is distanceinformation, and additional information is object type information, butnot limited thereto.

TABLE 2 Index (distance index, type index) Class (0, 0) Undefineddistance + Etc. (0, 1) Undefined distance + Sea (1, 2) Short distance +Geographical features (2, 2) Middle distance + Geographical features (3,2) Long distance + Geographical features (1, 3) Short distance + Vessel(2, 3) Middle distance + Vessel (3, 3) Long distance + Vessel (0, 4)Undefined distance + Rope

Table 2 shows an example of a method of representing locationinformation and type information according to an exemplary embodiment.Referring to Table 2, an index which includes a distance indexcorresponding to specific distance information and a type indexcorresponding to specific type information may be allocated to a classcorresponding to the specific distance information and the specific typeinformation. Here, each of the distance index and the type index may berepresented as a single value. As a non-limiting example, the index maybe represented in the form of (distance index, type index). In thiscase, a region or pixel labeled with a specific index in an image may beconsidered as being labeled with a specific distance index and aspecific type index. For example, referring to Table 2, a region orpixel labeled with an index (1, 3) may be considered as being labeledwith short distance and a vessel.

Referring to Table 2, distance indices may include a first distanceindex (0) corresponding to undefined distance, a second distance index(1) corresponding to short distance, a third distance index (2)corresponding to middle distance, and a fourth distance index (3)corresponding to long distance. Also, type indices may include a firsttype index (0) corresponding to etc., a second type index (1)corresponding to the sea, a third type index (2) corresponding togeographical features, a fourth type index (3) corresponding to avessel, and a fifth type index (4) corresponding to a rope.

TABLE 3 Index (distance index type index) Class (0 | 1, 0, 0, 0, 0, 0)Undefined distance + Etc. (0 | 0, 1, 0, 0, 0, 0) Undefined distance +Sea (1 | 0, 0, 1, 0, 0, 0) Short distance + Geographical features (2 |0, 0, 1, 0, 0, 0) Middle distance + Geographical features (3 | 0, 0, 1,0, 0, 0) Long distance + Geographical features (1 | 0, 0, 0, 1, 0, 0)Short distance + Vessel (2 | 0, 0, 0, 1, 0, 0) Middle distance + Vessel(3 | 0, 0, 0, 1, 0, 0) Long distance + Vessel (0 | 0, 0, 0, 0, 1, 0)Undefined distance + Rope (0 | 0, 0, 0, 0, 0, 1) Undefined distance +Crane

Table 3 shows another example of a method of representing locationinformation and type information according to an exemplary embodiment.Comparing Tables 2 and 3, it is possible to see that methods ofrepresenting type indices are different. Specifically, each index ofTable 3 may include a distance index represented as a single value and atype index represented as a set of a plurality of single values.

Referring to Table 3, each index may be represented to include onedistance index and a plurality of type indices. As a non-limitingexample, an index may be represented in the form of (distanceindex|first type index, second type index, . . . , nth type index). Inthis case, each of the plurality of type indices may correspond to thetype information of a specific object. For example, referring to Table3, when a specific region or pixel in an image corresponds to first typeinformation, a first type index corresponding to the first typeinformation may have a first value (e.g., 1), and other type indices mayhave a second value (e.g., 0). In this case, the specific region orpixel may be considered as being labeled with the first type index.

Table 3 shows a case in which type indices are represented as a set of aplurality of single values. On the contrary, an index may include adistance index represented as a set of a plurality of single values anda type index represented as a single value. Otherwise, an index mayinclude a distance index represented as a set of a plurality of singlevalues and a type index represented as a set of a plurality of singlevalues.

The above-described methods of representing distance information andtype information are only exemplary, and distance indices, type indices,indices, classes, etc. may be defined in a way different from themethods.

A case in which one index corresponds to one object or one piece ofdistance information has been described above, but one index maycorrespond to a plurality of objects or a plurality of pieces ofdistance information.

According to an exemplary embodiment, an index may be set to representobjects overlapping each other. FIG. 39 is a diagram relating to anindex which represents objects overlapping each other according to anexemplary embodiment. Referring to FIG. 39, an image 1000 may include aregion 1100 in which a vessel and a geographical feature overlap and aregion 1200 in which the vessel and the sea overlap.

As an example, when a single identification value is allocated to aclass as shown in Table 1, identification values may include a firstidentification value (e.g., 11) corresponding to both the vessel and thegeographical feature and a second identification value (e.g., 12)corresponding to both the vessel and the sea. In this case, referring toFIG. 39, in a region corresponding to the vessel in the image 1000, afirst identification value (e.g., 11) may correspond to the region 1100in which the vessel and the geographical feature overlap, and a secondidentification value (e.g., 12) may correspond to the region 1200 inwhich the vessel and the ocean overlap.

As another example, when classes correspond to indices including typeindices represented as a single value as shown in Table 2, the typeindices may include a first type index (e.g., 5) corresponding to both avessel and a geographical feature and a second type index (e.g., 6)corresponding to both a vessel and a sea. In this case, referring toFIG. 39, in the region corresponding to the vessel in the image 1000, anindex (e.g., (1, 5)) including the first type index may correspond tothe region 1100 in which the vessel and the geographical featureoverlap, and an index (e.g., (1, 6)) including the second type index maycorrespond to the region 1200 in which the vessel and the sea overlap.

As yet another example, when classes correspond to indices includingtype indices represented as a set of a plurality of single values asshown in Table 3, referring to FIG. 39, in the region corresponding tothe vessel in the image 1000, a first index (e.g., (1|0, 0, 1, 1, 0, 0))may correspond to the region 1100 in which the vessel and thegeographical feature overlap, and a second index (e.g., (1|0, 1, 0, 1,0, 0)) may correspond to the region 1200 in which the vessel and the seaoverlap. In this case, unlike the above-described examples related toTables 1 and 2, objects overlapping each other may be representedwithout adding an identification value or a type index.

Although the case in which the vessel and the geographical featureoverlap and the case in which the vessel and the sea overlap have beendescribed above, a case in which vessels overlap or the like may also berepresented in the same way. For example, a first vessel and a secondvessel overlap, and thus the first vessel may overlap at least a part ofthe second vessel behind the first vessel. In this case, an overlappingregion may not be labeled to correspond to the first vessel only but maybe labeled to correspond to both the first vessel and the second vessel.

Also, even when objects do not physically overlap, an sea region inwhich sea clutter is present may not be labeled to correspond to onlyone of sea clutter and the sea but may be labeled to correspond to bothsea clutter and the sea. This may also be applied to other cases such asa case of labeling a region in which backlight and a vessel overlap tocorrespond to both the backlight and the vessel.

According to an exemplary embodiment, an index may be set to representconnectivity between objects. Here, the connectivity may denote whetherthe objects are physically connected. Alternatively, the connectivitymay denote whether one of the objects is installed on the other.

FIG. 40 is a diagram relating to an index representing connectivitybetween objects according to an exemplary embodiment. Referring to FIG.40, an image 2000 may include a region 2100 corresponding to a craneinstalled on a vessel and a region 2200 corresponding to a craneinstalled on a port (or a geographical feature, a land).

As an example, when a single identification value is allocated to aclass as shown in Table 1, identification values may include a thirdidentification value (e.g., 13) corresponding to the crane installed onthe vessel and a fourth identification value (e.g., 14) corresponding tothe crane installed on the port. In this case, referring to FIG. 40, thethird identification value (e.g., 13) may correspond to the region 2100corresponding to the crane installed on the vessel in the image 2000,and the fourth identification value (e.g., 14) may correspond to theregion 2200 corresponding to the crane installed on the port.

As another example, when a class corresponds to an index including atype index represented as a single value as shown in Table 2, typeindices may include a third type index (e.g., 7) corresponding to thecrane installed on the vessel and a fourth type index (e.g., 8)corresponding to the crane installed on the port. In this case,referring to FIG. 40, in the image 2000, an index (e.g., (1, 7))including the third type index may correspond to the region 2100corresponding to the crane installed on the vessel, and an index (e.g.,(1, 8)) including the fourth type index may correspond to the region2200 corresponding to the crane installed on the port.

As yet another example, when a class corresponds to an index including atype index represented as a set of a plurality of single values as shownin Table 3, referring to FIG. 40, in the image 2000, a third index(e.g., (1|0, 0, 0, 1, 0, 1)) may correspond to the region 2100corresponding to the crane installed on the vessel, and a fourth index(e.g., (1|0, 0, 1, 0, 0, 1)) may correspond to the region 2200corresponding to the crane installed on the port. In this case, unlikethe above-described examples related to Tables 1 and 2, connectivitybetween objects may be represented without adding an identificationvalue or a type index.

Although the location at which the crane is installed has been describedabove as an example of connectivity between objects, connectivitybetween objects is not limited thereto. For example, a rope connected toat least one of a vessel or a port (or a geographical feature, a land)and the like may be represented in a similar way.

According to an exemplary embodiment, an index may be set to distinguishbetween the same type of objects. For example, separate identifiers maybe given to a plurality of vessels included in an image to distinguishthe plurality of vessels from each other. Here, the identifiers may beincluded in an index for representing object information. Also, givingan identifier to an object included in an image may denote labeling aregion or pixel of the image corresponding to the object with theidentifier.

According to an exemplary embodiment, whether to distinguish betweenobjects of the same type may vary according to distance information ofthe objects. FIG. 41 is a diagram relating to distinguishing betweenobjects of the same type on the basis of distance information accordingto an exemplary embodiment. Referring to FIG. 41, vessels 3100 and 3200corresponding to a first distance range which is adjacent to a port (ora geographical feature, a land) may be distinguished from each other,and a vessel 3300 corresponding to a second distance range which isfarther than the first distance range may not be distinguished. As anexample, when a vessel corresponds to the first distance range, anidentifier may be given to the vessel, and when the vessel correspondsto the second distance range, no identifier may be given to the vessel.As another example, different identifiers may be given to vesselscorresponding to the first distance range, and the same identifier maybe given to vessels corresponding to the second distance range. Here,the first distance range may be distance information closer than thesecond distance range.

According to an exemplary embodiment, an index may be set so that typeinformation with which an image is labeled may depend on distanceinformation of an object.

As an example, when an object corresponds to first distance information,the object may be labeled with first type information, and when theobject corresponds to second distance information, the object may belabeled with second type information. FIG. 42 is a set of diagramsrelating to an index which is set so that type information depends ondistance information according to an exemplary embodiment, andillustrates setting of an index so that whether to distinguish betweenthe upper surface and the side surface of a vessel may vary according todistance information of the vessel. Referring to the upper diagram ofFIG. 42, when the distance between a vessel and a port is a certaindistance or more (or corresponds to first distance information), aregion corresponding to a vessel 4100 in an image 4000 may be labeled asa vessel without distinguishing between the upper surface and the sidesurface. On the other hand, referring to the lower diagram of FIG. 42,when the distance between a vessel and a port is less than a certaindistance (or corresponds to second distance information indicating ashorter distance than the first distance information), a regioncorresponding to a vessel in an image 5000 may be labeled as an uppersurface 5100 or a side surface 5200. Here, the upper surface 5100 andthe side surface 5200 may be regions corresponding to the upper surfaceand the side surface of the vessel, respectively. Also, the uppersurface of the vessel may include a deck and a structure installed onthe deck (e.g., a bridge, a funnel, and a crane).

As another example, when an object corresponds to first distanceinformation, the object may be labeled with first type information, andwhen the object corresponds to second distance information, the objectmay be labeled with both the first type information and second typeinformation. Referring to the upper diagram of FIG. 42, when thedistance between the vessel and the port is a certain distance or more(or corresponds to first distance information), the region correspondingto the vessel 4100 in the image 4000 may be labeled as a vessel withoutdistinguishing between the upper surface and the side surface. On theother hand, referring to the lower diagram of FIG. 42, when the distancebetween the vessel and the port is less than a certain distance (orcorresponds to second distance information indicating a shorter distancethan the first distance information), a region corresponding to thevessel in the image 5000 may be labeled as at least one of the uppersurface 5100 or the side surface 5200, also labeled as the vessel at thesame time.

As yet another example, type information with which an image may belabeled when an object corresponds to first distance information may beat least partially different from type information with which the imagemay be labeled when the object corresponds to second distanceinformation. For example, only when the distance between the vessel anda port is less than a certain distance (or corresponds to first distanceinformation), may a specific object (e.g., a rope, a crane) beseparately labeled in the image.

Although the method of setting an index so that whether to distinguishbetween the upper surface and the side surface of a vessel may varyaccording to distance information of the vessel has been mainlydescribed above, an index may be set in a way other than theabove-described method so that type information with which an image islabeled may depend on distance information of an object.

Hereinafter, some embodiments of a method of labeling distanceinformation in an image are described.

According to an exemplary embodiment, distance information may belabeled on the basis of one region, one line, or one point in an image.Alternatively, distance information may be labeled on the basis of aspecific object in an image. A specific object which is a reference ofdistance information is referred to as a reference object below. As anon-limiting example, a reference object may be a vessel, a quay, ageographical feature, or the like. Alternatively, a reference object maybe an object in which a device (e.g., a camera) that generates an imageto be labeled is installed.

FIG. 43 is a diagram relating to distance information based on referenceobjects according to an exemplary embodiment. Referring to FIG. 43,distance information may include a first distance range 6200, a seconddistance range 6300 which is farther than the first distance range 6200,and a third distance range 6400 which is farther than the seconddistance range 6300 on the basis of a geographical feature which is areference object 6100. In this case, a region or pixel of an imagecorresponding to an object (e.g., a vessel, an obstacle) within thefirst distance range 6200 may be labeled with a distance indexindicating the first distance range 6200. Also, a region or pixel of animage corresponding to an object within the second distance range 6300and a region or pixel of an image corresponding to an object within thethird distance range 6400 may be labeled with a distance indexindicating the second distance range 6300 and a distance indexindicating the third distance range 6400, respectively.

According to an exemplary embodiment, labeling distance informationbased on a reference object may denote labeling the distance informationbased on a part of the reference object. For example, referring to FIG.43, distance information may be labeled based on a boundary of thegeographical feature which is the reference object. In addition,distance information may also be labeled on a region or point of areference object.

According to an exemplary embodiment, a reference object may be presentin a specific region of an image. As an example, a reference object maybe present in a lower portion of an image. As another example, areference object may be present in at least one of upper, lower, left,or right portions of an image.

According to an exemplary embodiment, distance information may include aplurality of distance ranges defined between first and second referencesin an image. As an example, at least one of the first or secondreferences may be a reference object. As another example, the firstreference may be a reference object, and the second reference may be aspecific region, a specific line, or a specific point in an image.

According to an exemplary embodiment, distance information may be setfor a drivable region (e.g., a sea, a road) in an image. For example, adrivable region may be divided into a plurality of regions, anddifferent pieces of distance information (e.g., short distance, middledistance, long distance) may be set for each of the plurality ofregions. In this case, an object located on any one of the plurality ofregions may be labeled with a distance index corresponding to distanceinformation set for the any one of the plurality of regions.

According to an exemplary embodiment, the boundary between adjacentdistance ranges among the plurality of distance ranges may be set byconsidering a reference object. For example, the boundary may be set byconsidering at least one of the shape or boundary of a reference object.Referring to FIG. 43, the boundary between the first distance range 6200and the second distance range 6300 may be set by considering theboundary of the geographical feature which is the reference object.

According to an exemplary embodiment, when an object is located over aplurality of distance ranges, the object may be labeled with a distanceindex corresponding to any one of the plurality of distance ranges. Forexample, when one portion of the object is present within a firstdistance range and the other portion is present within a second distancerange, the whole object may be labeled with a distance indexcorresponding to a closer one of the first and second distance indices.

Although the case in which distance information includes three distanceranges has been mainly described above, the distance information is notlimited thereto. Distance information may include two distance ranges orfour or more distance ranges. Also, the setting of distance ranges mayvary by considering the installation position, installation direction,specifications (e.g., FOV), etc. of a camera. For example, the settingof distance ranges in a first image may differ from the setting ofdistance ranges in a second image.

According to an exemplary embodiment, whether to reflect an object in anobstacle map may vary according to distance information of the object.For example, only an object corresponding to a specific distance rangemay be reflected in an obstacle map. Here, the specific distance rangemay be a distance range indicating a relatively close distance among aplurality of distance ranges which may correspond to an object.

According to an exemplary embodiment, when an object is present within afirst distance range, the object may not be reflected in an obstaclemap, and when an object is present within a second distance range, theobject may be reflected in the obstacle map. Here, the second distancerange may be closer than the first distance range.

According to an exemplary embodiment, a first object present within afirst distance range may not be reflected in an obstacle map, and asecond object present within a second distance range may be reflected inthe obstacle map. Here, the second distance range may be closer than thefirst distance range.

According to an exemplary embodiment, whether to track an object mayvary according to distance information of the object. Here, tracking anobject may denote storing and/or monitoring information on an objectsuch as the type, location, direction, velocity, and speed of theobject.

According to an exemplary embodiment, when an object is present within afirst distance range, the object may not be tracked, and when the objectis present within a second distance range, the object may be tracked.Here, the second distance range may be closer than the first distancerange.

According to an exemplary embodiment, a first object present within afirst distance range may not be tracked, and a second object presentwithin a second distance range may be tracked. Here, the second distancerange may be closer than the first distance range.

According to an exemplary embodiment, location information and typeinformation of an object may be acquired together through one model. Forexample, output data including location information and type informationcorresponding to a pixel of an image may be acquired by inputting theimage to the model. As a non-limiting example, the model may be anartificial neural network.

According to an exemplary embodiment, type information of an object maybe acquired through a first model, and location information of theobject may be acquired through a second model. For example, an image maybe input to the first model to acquire first output data including typeinformation corresponding to a pixel of the image, and at least one ofthe image or the first output data may be input to the second model toacquire second output data including location information correspondingto the pixel of the image. As a non-limiting example, at least one ofthe first model or the second model may be an artificial neural network

According to an exemplary embodiment, in an inference operationemploying a model such as an artificial neural network, at least some ofa plurality of pieces of object information outputted from the model maybe handled differently from output information outputted from the modelsuch as being changed or ignored on the basis of other pieces of theobject information. For example, when an image is inputted to the modelto acquire location information and type information, the locationinformation may not be taken into consideration according to the typeinformation.

According to an exemplary embodiment, even when a region or pixel of animage corresponding to a sea in training data is labeled with a distanceindex corresponding to an undefined distance (e.g., Table 2 or Table 3),in an inference operation, a region or pixel of an image correspondingto a sea may be labeled with a distance index other than the undefineddistance (e.g., a short distance, a middle distance, a long distance).In this case, distance information of the region or pixel of the imagecorresponding to the sea may be considered the undefined distanceregardless of which distance index is outputted through inference.

According to an exemplary embodiment, a model (e.g., an artificialneural network) which outputs information on at least one of the bow orstern of a vessel may be provided. For example, the model may receive animage and output location information of at least one of the bow orstern of a vessel included in the image. Here, the model may output thelocation information by displaying at least one of the bow or stern inthe image. Alternatively, the model may output coordinate valuescorresponding to at least one of the bow or stern in the image.

According to an exemplary embodiment, the model which outputsinformation on at least one of the bow or stern of a vessel may betrained with training data including an input image and locationinformation of at least one of the bow or stern of a vessel included inthe input image. For example, the model may be trained on the basis ofoutput data outputted from the model receiving the input image and thelocation information included in the training data.

According to an exemplary embodiment, a model which outputs informationon a side line of a vessel may be provided. For example, the model mayreceive an image and output information on a side line of a vesselincluded in the image. Here, the model may output the information on theside line by displaying the side line in the image. Alternatively, themodel may output the information on the side line by displaying one ormore points corresponding to the side line in the image. Alternatively,the model may output coordinate values corresponding to one or morepoints corresponding to the side line in the image.

According to an exemplary embodiment, the model which outputsinformation on a side line of a vessel may be trained with training dataincluding an input image and information on a side line included in theimage. For example, the model may be trained on the basis of output dataoutputted from the model receiving the input image and the informationon the side line included in the training data.

According to an exemplary embodiment, a model which outputs an anglebetween a plurality of objects may be provided. For example, the modelmay receive an image and output the angle between a plurality of objectsincluded in the image. As an example, the angle between a plurality ofobjects may be the angle between a vessel and a quay (or a geographicalfeature, a land). As another example, the angle between a plurality ofobjects may be the angle between a vessel and another vessel.

According to an exemplary embodiment, the model which outputs the anglebetween a plurality of objects may be trained with training dataincluding an input image and the angle between a plurality of objectsincluded in the input image. For example, the model may be trained onthe basis of output data outputted from the model receiving the inputimage and the angle included in the training data.

According to an exemplary embodiment, a model which converts an imageinto a bird's eye view may be provided. For example, the model mayreceive a perspective-view image and output a bird's-eye view image.

According to an exemplary embodiment, the model which converts an imageinto a bird's eye view may be trained with training data including aninput perspective-view image and an image obtained by converting theinput image into a bird's eye view. For example, the model may betrained on the basis of output data outputted from the model receivingthe input perspective-view image and the image obtained by convertingthe input image into a bird's eye view.

The methods according to exemplary embodiments may be implemented in theform of program instructions, which are executable by various computermeans, and stored in a computer-readable recording medium. Thecomputer-readable recording medium may include program instructions,data files, and data structures independently or in combination. Theprogram instructions stored in the computer-readable recording mediummay be specially designed and constructed for the exemplary embodimentsor may be well-known to those of ordinary skill in the computer softwarefield. Examples of the computer-readable recording medium may includemagnetic media, such as a hard disk, a floppy disk, and a magnetic tape,optical media, such as a compact disc read-only memory (CD-ROM) and adigital versatile disc (DVD), magneto-optical media, such as a flopticaldisk, and hardware devices, such as a ROM, a random access memory (RAM),and a flash memory, which are specifically constructed to store andexecute program instructions. Examples of the program instructionsinclude high-level language code executable by a computer using aninterpreter or the like as well as machine language code made by acompiler. The hardware devices may be configured to operate as one ormore software modules or vice versa to perform operations of theexemplary embodiments.

According to exemplary embodiments of the present invention, it ispossible to sense surroundings using a neural network which performsimage segmentation.

According to exemplary embodiments of the present invention, it ispossible to train a neural network for situation awareness.

Effects of the present invention are not limited to the effectsdescribed above, and other effects that are not mentioned above may beclearly understood by those of ordinary skill in the art to which thepresent invention pertains from this specification and the appendeddrawings.

Since various substitutions, modifications, and variations can be madefrom the above-described present invention without departing from thetechnical spirit of the present invention by those of ordinary skill inthe art, the present invention is not limited to the above-describedembodiments or the accompanying drawings. Also, the exemplaryembodiments described herein are not separately applied, and all or someof the exemplary embodiments may be selectively combined so that variousmodifications can be made. Further, operations constituting eachexemplary embodiment may be used separately from or in combination withoperations constituting another exemplary embodiment.

What is claimed is:
 1. A method for situation awareness, the methodcomprising: preparing a neural network trained by a learning set,wherein the learning set includes a plurality of maritime images andmaritime information including object type information which includes afirst type index for a vessel, a second type index for a water surfaceand a third type index for a ground surface, and distance levelinformation which includes a first level index indicating that adistance is undefined, a second level index indicating a first distancerange and a third level index indicating a second distance range greaterthan the first distance range, wherein pixels, in the plurality ofmaritime images, being labeled with the first type index, are labeledwith the second level index or the third level index, and pixels, in theplurality of maritime images, being labeled with the first level index,are labeled with the second type index or the third type index;obtaining a target maritime image generated from a camera, the camerabeing installed on a port or a vessel and monitoring surroundingsthereof; and determining a distance of a target vessel in thesurroundings based on the distance level index of the maritimeinformation being outputted from the neural network which receives thetarget maritime image and having the first type index.
 2. The method ofclaim 1, wherein the object type information further includes a fourthtype index for a top surface of a vessel and a fifth type index for aside surface of a vessel, and wherein pixels, in the plurality ofmaritime images, being labeled with the first type index, are labeledwith the third level index, and pixels, in the plurality of maritimeimages, being labeled with the fourth type index or the fifth typeindex, are labeled with the second level index.
 3. The method of claim1, wherein the object type information further includes a fourth typeindex for a top surface of a vessel and a fifth type index for a sidesurface of a vessel, and wherein pixels, in the plurality of maritimeimages, being labeled with the first type index and the second levelindex, are labeled with one of the fourth type index and the fifth typeindex.
 4. The method of claim 1, wherein the object type informationfurther includes a fourth type index for a crane, and wherein pixels, inthe plurality of maritime images, being labeled with the fourth typeindex, are labeled with the first type index or the third type index. 5.The method of claim 1, wherein the method further comprises: detecting awater surface in the surroundings, using the neural network, from thetarget maritime image, according to the object type information of themaritime information being outputted from the neural network based onthe target maritime image and having the second type index; anddetermining a distance to the detected water surface as an undefineddistance.
 6. The method of claim 5, wherein the determining comprisesignoring the distance level information of the maritime informationbeing outputted from the neural network based on the target maritimeimage and having the second type index.
 7. The method of claim 1,wherein the maritime information further includes an object identifier,and wherein pixels, in the plurality of maritime images, being labeledwith the first type index and the second level index, are labeled withthe object identifier.
 8. The method of claim 7, wherein the pixelsbeing labeled with the first type index and the second level indexinclude one or more first pixels and one or more second pixels, and theobject identifier labeled to the first pixels is different from theobject identifier labeled to the second pixels.
 9. The method of claim1, wherein the method further comprises: generating an obstacle mapbased on the measured distance of the vessel, wherein the target vesselis reflected on the obstacle map when the outputted distance level indexcorresponds to the second level index.
 10. The method of claim 1,wherein the determining comprises obtaining distance information of thetarget vessel based on at least one of a location of the target vesselon the target maritime image and a position information of the camera,and wherein the distance of the target vessel is determined furtherbased on the obtained distance information.
 11. The method of claim 10,wherein the obtained distance information is represented by a distancevalue, wherein the determining further comprises determining an errorbased on whether the obtained distance information is included in theoutputted distance level index, and wherein the distance of the targetvessel is determined further based on the error.
 12. The method of claim11, wherein when the error occurs, the distance of the target vessel isdetermined as a distance range corresponding to the outputted distancelevel index, and when the error does not occur, the distance of thetarget vessel is determined as a distance range from the distance valueto a maximum value of the distance range corresponding to the outputteddistance level index.
 13. A computer-readable recording medium in whicha program for performing the method of claim 1 is recorded.
 14. A devicefor situation awareness, the device comprising: a memory for storing aneural network trained by a learning set, wherein the learning setincludes a plurality of maritime images and maritime informationincluding object type information which includes a first type index fora vessel, a second type index for a water surface and a third type indexfor a ground surface, and distance level information which includes afirst level index indicating that a distance is undefined, a secondlevel index indicating a first distance range and a third level indexindicating a second distance range greater than the first distancerange, wherein pixels, in the plurality of maritime images, beinglabeled with the first type index, are labeled with the second levelindex or the third level index, and pixels, in the plurality of maritimeimages, being labeled with the first level index, are labeled with thesecond type index or the third type index; and a controller configuredto: obtain a target maritime image generated from a camera, the camerabeing installed on a port or a vessel and monitoring surroundingsthereof; and determine a distance of a target vessel in the surroundingsbased on the distance level index of the maritime information beingoutputted from the neural network which receives the target maritimeimage and having the first type index.